XPLAB's testphilosophy for Quality Assurance                  17.04.2004
========================================================================

With the distribution you got a test suite. This is a directory
containing all testcases which are to be used to develop XPLAB and ensure
the consistency of the development. They are originated by unsatisfying
results and therefore offer sensitive tests. They may show you XPLAB's
strength and weakness. 

XPLAB offers a GUI in opposite to PLAB, my first CLI-based
program. Instead of many options XPLAB uses the configuration file. This
is created using the GUI "File/preferences". The testcases however avoid
the GUI and run in the batch mode, which is called with the option "-Y".


Test script
========================================================================

A test script 'xrun_tp', written in PERL, runs all testcases and prints
the results to stdout. For a better overview the testcases are collected
in testlists.

There is a file 'last_result' which contains the output of the
testscript, generated before uploading the distribution. It is for
convenience only to have a reference for your own results. Generation
is './xrun_tp > last_result'.

The test results produced by `xrun_tp' can be rather big if they differ
from their output references, therefore it's a good idea redirecting to
a file like `./xrun_tp > result'.

Test results can be:-
	PASS	no differences in validated and report files
	WARN	differences in report files
	FAIL	differences in validated files
	COPY	like WARN, report files will be updated

If there are testcases missing PASS, the temporary files created by PLAB
will not be deleted and the script outputs the used option string.


Test script options
========================================================================

The test script offers some few options, which are used for maintaining
the testcases and for development. If a testnumber is input, the script
runs only the according test.

-h  Show all options.

-l  List all tests.

-u  Update informal reference files causing warnings by copying the last 
    created output to the reference. This is not the case if the test
    fails. If a test, delivered with the distribution, fails, usually
    code or test has to be fixed. The reference files must be validated
    manually.

The following options are not subject for common tests and are not
maintained. They are used for development and should not be used.

-k  Suppress deletion of temporary files.


Test cases
========================================================================

The testcases are grouped in testlists, however, the grouping is not
important. Important is the fact, that all testcases in the distribution
are delivered with PASS results. If that is not the case, please send me
an email. The common data flow of a testcase is as follows:-

                                               references
     input        config        output         (*.TC-OUT)      test   
   (*.TC-IN)       file       (temporary)      ==========     result  
   =========        |         ===========          |          ====== 
(image file)-+      V     +-->(text file)----+     V
             +-->[XPLAB]--+-->(report file)--+-->[diff]--->{PASS...FAIL}
  (database)-+            +-->(other files)--+


The tests are mainly for pattern recognition. Word space decision and
layout is made but not in the focus. Some tests offer patterns which
cannot recognized correctly yet. They show the limits of XPLAB, which I
think about as important as the good cases. However, I don't like to
discuss competitions among wrong results. Its not very helpful to
discuss how bad wrong results are. The consequence is, any wrong
recognition can be replaced by another wrong recognition. That is
important for improvements or when new algorithms will be introduced. Of
course, correct results never can be replaced by incorrect ones, even if
there is a mix of both of them.

Test input files have the extension "TC-IN" and expected test output
files have the extension "TC-OUT" used as references. The expected
output files are validated XPLAB results. If a test output does not
differ with the references the test result is set to PASS, otherwise the
output has to be validated again or code has to be fixed. Some sort of
files ('report files') basically can have any content and are used for
test completeness. Report files never produce a FAIL if there are
differences but a WARN.

Changes of the validated files to adapt new results should be made with
caution. Some validated files still contain symbols, which can be
improved by new methods in the future.


Test image types
======================================================================

Test files `SequenceTest*' are not designed for certain results but for
high sensibility of methods and indication of code changes. Some of them
cannot have a certain correct result at all. They should show the
uncertainties of the algorithms. This is also true for the files
`SeparationTest*'. Therefore it is always possible to fit the validated
files to avoid errors or warnings, respectively, by running the test,
without loosing any important information.

Test files `ClusterTest*' mainly depends of the cropping features of the
used clustering method. Small differences in the results are possible
because of the recursion due to several possible solutions for
clustering which are acceptable.

Test files `screen*' and `spanning*' contain undisturbed original
symbols. They were derived from screen dumps. That sort of input
completes the field of pattern matching because it raises some problems
in the area of layout and competition among candidates, not easily
triggered by pattern with noise. The database for these patterns can be
kept small with one sample per class.

All other test image files, not mentioned above, contain normal patterns
delivered from a scanner (300dpi).

Just display the input image files with your preferred image rendering
tool.


Personal tests
======================================================================

Usually the delivered tests run as expected ('./xrun_tp' 'PASS'es all
tests). But you want your own tests with your own input, of course. The
best is to put new testinput in a separate directory and perform their
test in that place. This avoids confusion with the public package. If
there is a flaw, the methods have to be improved and after success the
image, which causes the flaw, is taken as test input.

The testscript helps you defining your own tests. Even if you are not
familiar with PERL's scripting language, you should be able to add some
own tests. The preferred way is to name new test files like the existing
ones, e.g. `<filename>.TC-IN', as input, `<filename>.rpt.TC-OUT', the
report file and `<filename>.txt.TC-OUT', the validated text file. To get
the test output files, run XPLAB (see `XPLAB Tutorial' on it's homepage)
and rename the result files properly.


What to do with `WARN'ed tests?
======================================================================

Warnings, caused by different report file contents, should be checked
and normally are to be adjusted to the new one (just copy or rename
it). Report files only give some internal information and are sensitive
to code changes.


What to do with `FAIL'ed tests?
======================================================================

As I developed the test cases for passing conditions, failed test cases
are severe. You could try your own solution and improve test or code. On
success please let me know.

Alternatively isolate the cause (cutting out images to make them small)
and send it to me together with XPLAB's version number:

	elmar.sack@mnet-online.de

There could be another, really annoying fault, that is, when XPLAB
crashes. Even if I tested XPLAB for a long time, I cannot give any
guarantee not to fail.

======================================================================




