This page contains additional information to the article "Empirical Testing of a Weight Constraint Rule Based Configurator" by Juha Tiihonen , Timo Soininen, Ilkka Niemelä, and Reijo Sulonen
This page is not self-exlanotory: you should have the article to understand fully the specifics on this page. Download and extract empirical.zip to get the PCML, WCRL, and BCRL configuration models of the test cases, test results, run logs and the modified smodels version 2.26.
For each configuration model, we generated 100 test cases with 2 requirements, 100 test cases with 4 requirements, etc, for each even number of requirements up to the number of groups. The random requirements in a test case were expressed as a Smodels compute statement. A Java based test generator and driver was used to generate and execute the test cases. In the following a case set refers to the set of requirements with a specific number of requirements related to a specific configuration model. For example a case set is formed by the 100 test cases related to the ESVS configuration model with 4 requirements.
The main directory contains two files and six directories:
| File | Contents |
| PCML to WCRL.txt | PCML to WCRL translation times result |
| Combination.xls | Combined results in MS Excel format |
There is a test case directory for each tested configuration model and a directory for the modifed version of smodels 2.26. used in the tests.:
| Directory | Contents |
| ESVS | Directory for the ESVS compressor |
| FS | Directory for the FS compressor |
| FX | Directory for the FX compressor |
| Machine | Directory for the Vehicle configuration model (The model was named Machine for the test runs) |
| smodels | Directory for the modified smodels for Windows/DOS command
prompt and the modified source files. Umodified Smodels is publicly available
at http://www.tcs.hut.fi/Software/smodels/.
|
A test case directory has files with the following extensions
| File extension | Description |
| .tst |
Single test case (a set of requirements as Smodels compute statement). Test cases are named as Name_Nreqs_Nth.tst. For example esvs100_10_34.tst is from esvs100 (named so because the set tests the ESVS configuration model and has 100 test cases per number of requirements). The test case has 10 requirements (Nreqs=10) and it is the 34th test case with 10 requirements (Nth=34). |
| .csd |
Case set definition file: lists all test cases of the case set. For example, Esvs100_10.csd defines the test cases to run with 10 requirements for test set esvs100 |
| .csa | A .csa file lists case sets in the order in which they are run |
| .out |
Results of all case setes. The results were gathered during runtime and calculations were made. The calculated results are represented as delimited ASCII. .out files are easy to import to spreadsheet programs like Microsoft Excel. Successful cases are referred to as "OK" in the headers .out files. Failures are referred to as "Fail". |
| .grp | group definitions. XML format understood by the Java based test generator. Details are given below. |
| .lpi | WCRL form of the configuration model (logic program input for lparse) |
| .lpo | BCRL form of the configuration model (logic program output by lparse) |
| .cfg | PCML form of the configuration model |
| *Groupgenerator.xls |
*Groupgenerator.xls Excel spreadsheet for generating the .grp file. For example ESVSGroupGenerator.xls generated the group file for ESVS configuration model. Was constructed manually from the atom list for each configuration model. Includes macros, but no viruses. |
A test case directory has two result subdirectories for results of different runs.
| Directory | Description |
| res100_1 | Subdirectory for results of test sets with 100 test cases and the configuration task of finding the first configuration |
| res100_all | Subdirectory for results of test sets with 100 test cases and the configuration task of finding all the configurations |
A result subdirectory has files with the following extensions
| Extension | Description |
| .csr |
Case set results as a run log. For example, esvs100_14.csr represents esvs100 results for 14 requirements. Formed by capturing the command prompt output through Java's mechanisms. |
| .out | test results of all case sets. Delimited ASCII format. Delimiter is ';'. Includes heading rows, too. In one model the headers may be aligned incorrectly. They have been repaired in the Excel version. |
| .xls | .out imported to Microsoft Excel and sorted in ascending order by number of requirements. |
An example portion of a .csr file is shown below for running the test case esvs100_14_1.tst (some line breaks added manually)
% Processing csd: c:/projects/CfgTester/ESVS/esvs100_14.csd % Start at:Sat Mar 02 20:05:26 GMT+02:00 2002 CASE:c:/projects/CfgTester/ESVS/esvs100_14_1.tst C:\projects\CfgTester>c:\p\bin\lparse c:/projects/GardnerDenver/esvs.lpi c:/projects/CfgTester/ESVS/esvs100_14_1.tst | c:\p\bin\smodelsjt -pn -nth 1000000000 0 smodels version 2.26. Reading...done Answers found: 0 False Duration 0.300 Number of choice points: 0 Number of wrong choices: 0 Number of atoms: 1223 Number of rules: 1258 Number of picked atoms: 0 Number of forced atoms: 0 Number of truth assignments: 415 Size of searchspace (removed): 0 (0) TotalRunTime_ms=360 CaseResult c:/projects/CfgTester/ESVS/esvs100_14_1.tst: totalRuntime_ms=360, answers=0, fullAnswers=0, moreAnswers=0, smodelsDuration_ms=300
An intermediate version of our article explained the test case generation process more procedurally than the submitted version. It explains the format of .grp files with more detail than the paper.
A requirement group (group for brevity) represents a set of potential requirements that a user could make. For example, a group could represent selection of power property for a motor, or selection of a part realization for cooling system in a compressor. The maximum number of requirements generated from a group is defined by maximum cardinality. Minimum cardinality specifies a minimum number of requirements generated from the group, if the group is chosen to generate a requirement. When generating a test case, a random group is selected to generate a requirement (or minimum number of requirements specified by the cardinality, if the group has not been chosen before). A group can be chosen again, if it's maximum cardinality has not been reached yet. Group selection is repeated until the desired number of requirements has been generated.
Each group has requirement items. When a group is selected to generate
a requirement, a requirement item is randomly selected. To facilitate requirement
generation e.g. for parts with symmetry breaking, items in the groups are actually
arranged into queues. This guarantees, for example, that the first individual
b1 of type B is required before the second individual of type B, b2. A requirement
is generated from a group by choosing randomly one queue and picking it's head
as the generated requirement.
The smodels direcory contains the modified smodels executable and source files. The source files in the smodels directory replace the source files of the original smodels 2.26 distribution.
Smodels was modified by implementing two command line switches, using which output of found configurations was suppressed. Suppressing the output was needed to avoid the configuration task to become IO bound due to relatively large number of atoms printed for each configuration. Instead, the number of found configurations was reported. No other switches were given to lparse or smodels.
The newly implemented smodels switches are -nth
N to print only every N:th stable model found and -pn
to print only the number of the stable model instead of the full stable model.
The modifications are only partial for optimization use of smodels. -pn
and -nth switches may not work
correctly when smodels is given an optimization task. Optimization was not used
in our tests.
The modified files are smodels.h, smodels.cc and main.cc. the executable is
called smodelsjt.exe.
The PCML to WCRL translation uses type-checked and validated Java data structures parsed into an internal representation. For example, effective part definitions and property domains have their corresponding object instances. (Refinementsmay change inherited part defintions. Effective definitions are those where refinement has been taken into account).
In constraint translation, part references, property references and configuration
predicates as well as constants are recognized from the parse tree.
The following details may be helpful when browsing through the WCRL form of configuration models as the traslator itself is not publicly available.
The sentences in a generated WCRL program are in the following order: 1) ontological definitions, 2) component type hierarchy, 3) compositional structure rules 4) component individuals, 5) properties, and 6) constraints.
In the compilation process, an internal symbol table is created to associate names of objects in the PCML to those used in the resulting WCRL. Component type names of PCML are prefixed in WCRL with "compT_", parts with "part_", and properties with "prop_". If many types have properties or parts with the same name, _N postfixes (where N is an integer) are added as needed to give each property a unique name.