Firebird test suite

Introduction

• TCS
  Released by Borland with InterBase code along with 390 tests. Tests are written in mixture of shell script, arcane TCS commands, environment variables (both OS and TCS), C/C++, ESQL etc. with use of various external tools (all are standard, free or part of TCS suite), all packed in single text BLOB field in FB database. TCS is hard to operate, and writing new tests require almost the same skills (or more in certain areas) for QA people as for core developers. The entry barrier for any apprentice QA developer is very high.

  Although it's still in use, this test harness is deprecated, and all QA development is focused on new system.

• QMTest
  In our search for TCS replacement, we evaluated many options, and QMTest came out as the best one. QMTest is open source, general-purpose, cross-platform software testing tool written in Python. It has both, nice web-based GUI and CLI interface, and supports all features we need from such system. Of course, it doesn't fit these needs perfectly, but QMTest is designed to be easily extended or tailored for any special requirements.

  We wrote some extensions to QMTest that simplify creation of black-box test cases for Firebird engine. These extensions provide built-in support for test-database creation and initialisation, ISQL script execution, allow us to write even very complex tests in Python with use of KinterbasDB connectivity package for Firebird, and much more. Test cases could be created in QMTest web GUI, in our own test editor (preferred option), or by any text or XML editor as they're stored in XML.

Installation

1. Firebird, as running tests without it doesn't make sense :).
2. Python 2.4
   If you're using Windows, you may prefer Python from ActiveState.
3. KinterbasDB for Python 2.4.
   There are three different versions of KintebasDB for Firebird 1.0, 1.5 and 2.0 ! Be sure that you have installed the right one.
4. QMTest.
5. Subversion client.

All software listed here is open source. Installation of these packages is straightforward, and shouldn't cause you any trouble.

Next you would need the test suite for Firebird, that's created as QMTest test database. A test database stores tests, test suites, resources and QMTest extensions. QMTest stores a test database in a single directory, which may include many files and subdirectories.

You may have and use many QMTest Test Databases (see below), but only one Test Database is primary, and contains all our tests, extensions etc. This main database is called Test Repository. This Repository is located in our Subversion repository.

The path to the test suite is:
https://firebird.svn.sourceforge.net/svnroot/firebird/qa/trunk/testsuite

The Test Repository contains our QA DataBase Manager called qadbm. It's a multipurpose utility written in Python. It's a CLI utility with it's own command interpreter and integrated help. To list all qadbm commands, use the help command. You can get command specific help with help command-name.
All qadbm commands are case sensitive !

Building "runnable" test suite

To build platform/engine specific test suite, follow next steps:

1. Start the qadbm utility.
2. If you have not started qadbm from directory where your Local Test Repository resides, use REPOSITORY command to specify its directory.
   For example: repository /home/fbqa
3. Set the desired platform with PLATFORM command. Supported values are "Windows, Linux, HP-UX, FreeBSD, Solaris and Sinix-Z".
   For example: platform Windows
4. Set the target engine version.
   For example: target 1.5.2
5. Initialize a new QMTest test database with CREATE command.
   For example: create /home/fbtests
6. Copy all tests for specified platform and engine version into new QMTest test database with COPY command.
   For example: copy /home/fbtests

Configuration

• temp_directory : Directory for temporary files
• server_location : Hostname where Firebird is located (empty for local protocol access, hotname: for TCP/IP access)
• database_location : Directory where test databases are created
• backup_location : Path to database backup files. These files are part of the Test repository (subdirectory 'fbk')
• isc4_path : Path to security database, including filename
• user_name : Firebird user used by tests to connect/create database
• user_password : Firebird password for user_name user
• isql_path : Path to isql, including filename
• gstat_path : Path to gstat, including filename
• gsec_path : Path to gsec, including filename
• gbak_path : Path to gbak, including filename
• nbackup_path : Path to nbackup, including filename
• gfix_path : Path to gfix, including filename
• gpre_path : Path to gpre, including filename

Sample context setup for Firebird 2.0 on Linux:

temp_directory=/var/tmp/
server_location=localhost:
database_location=/var/tmp/ 
backup_location=/home/testsuite/fbk/
isc4_path=/opt/firebird/security2.fdb
user_name=SYSDBA
user_password=masterkey
isql_path=/opt/firebird/bin/isql
gsec_path=/opt/firebird/bin/gsec
gstat_path=/opt/firebird/bin/gstat
gbak_path=/opt/firebird/bin/gbak
nbackup_path=/opt/firebird/bin/nbackup
gfix_path=/opt/firebird/bin/gfix
gpre_path=/opt/firebird/bin/gpre

Running tests

It's convenient for Windows users to create a simple .BAT file with next command:

c:\Python23\python c:\python23\scripts\qmtest.py %1 %2 %3 %4 %5 %6 %7 %8 %9

The qmtest command has next structure:

qmtest [ option ...] command [ command-option ...] [ argument ...]

Options

These options can be used with any QMTest command, and must precede the command name on the command line.

All options are available in a "long form" prefixed with "--" (two hyphens). Some options also may be specified in a "short form" consisting of a single hyphen and a one-letter abbreviation. Short-form options may be combined; for example, -abc is equivalent to -a -b -c.

-D path, --tdb path

Use the test database located in the directory given by path. This flag overrides the value of the environment variable QMTEST_DB_PATH. If neither this flag nor the environment variable is specified, QMTest assumes that the current directory should be used as the database.

-h, --help

Display help information, listing commands and general options for the qmtest command.

--version

Additional options are available for specific commands; these are presented with each command. Options specific to a command must follow the command on the command line. Specify the --help (-h) option after the command for a description of the command and a list of of available options for that command.

Note:
Because Firebird test suite is stored in QMTest database, you must either run qmtest from directory where it's stored, or use the -D or --tdb option.

Commands

There are many qmtest commands (read QMTest documentation for complete list of supported commands), but only next three might interest you:

gui command

Start the graphical user interface.

qmtest gui [ option ...]

The qmtest gui starts the graphical user interface. The graphical user interface is accessed through a web browser. You must have a web browser that supports JavaScript to use the graphical interface. QMTest has been tested with recent versions of Internet Explorer and Mozilla. Other web browsers may or may nor work with QMTest.

The gui command accepts these options:

-A address, --address address

Bind the server to the indicated internet address, which should be a dotted quad. By default, the server binds itself to the address 127.0.0.1, which is the address of the local machine. If you specify another address, the server will be accessible to users on other machines. QMTest does not perform any authentication of remote users, so you should not use this option unless you have a firewall in place that blocks all untrusted users.

-c name=value, --context name=value

For details about this option, see the description of the qmtest run command.

-C file, --load-context file

For details about this option, see the description of the qmtest run command.

--daemon

Run the QMTest GUI as a daemon. In this mode, QMTest will detach from the controlling terminal and run in the background until explicitly shutdown.

-j count, --concurrency count

For details about this option, see the description of the qmtest run command.

--no-browser

Do not attempt to start a web browser when starting the GUI. QMTest will still print out the URL at which the server can be accessed. You can then connect to this URL manually using the browser of your choice.

-O file, --outcomes file

For details about this option, see the description of the qmtest run command.

--pid-file path

Specify the path to which the QMTest GUI will write its process ID. This option is useful if you want to run QMTest as a daemon. If this option is not provided, no PID file is written. If you specify this option, but path is the empty string, QMTest will check the .qmrc configuration file for a pid-file entry. If there is no such entry, QMTest will use an appropriate platform-specific default value.

--port port

Specify the port on which the QMTest GUI will listen for connections. If this option is not provided, QMTest will select an available port automatically.

-T file, --targets file

For details about this option, see the description of the qmtest run command.

Note:
The qmtest gui is the most comfortable way to run tests and to analyze the results, and to perform many test-suite related tasks (analyze stored run results, edit or create new tests, set test expectations, set context values etc.). You don't need to use any additional options, but using -C option to load prepared context values and -P option to set specific port number instead random one come very handy.

Run tests or test suites.

qmtest run [ option ...] [test-name | suite-name...]

The qmtest run command runs tests and displays the results. If no test or suite names are specified, QMTest runs all of the tests in the test database. If test or suite names are specified, only those tests or suites are run. Tests listed more than once (directly or by inclusion in a test suite) are run only once.

The run command accepts these options:

-c name=value, --context name=value

Add a property to the test execution context. The name of the property is name, and its value is set to the string value. This option may be specified multiple times.

-C file, --load-context file

Read properties for the test execution context from the file file.

The file should be a text file with one context property on each line, in the format name=value. Leading and trailing whitespace on each line are ignored. Also, blank lines and lines that begin with "#" (a hash mark) are ignored as comments.

This option may be specified more than once, and used in conjunction with the --context option. All of the context properties specified are added to the eventual context. If a property is set more than once, the last value provided is the one used.

If this option is not specified, but a file named context exists in the current directory, that file is read. The properties specified in this file are processed first; the values in this file can be overridden by subsequent uses of the --context option on the command line.

-f format, --format format

Control the format used when displaying results. The format specified must be one of full, brief, stats, batch, or none. The brief format is the default if QMTest was invoked interactively; the batch format is the default otherwise. In the full format, QMTest displays any annotations provided in test results. In the brief mode only the causes of failures are shown; detailed annotations are not shown. In the stats format, no details about failing tests are displayed; only statistics showing the number of passing and failing tests are displayed. In the batch mode, the summary is displayed first, followed by detailed results for tests with unexpected outcomes. In the none mode, no results are displayed, but a results file is still created, unless the --no-output option is also provided.

-j count, --concurrency count

Run tests in multiple count concurrent processes on the local computer. On multiprocessor machines, the processes may be scheduled to run in parallel on different processors. QMTest automatically collects results from the processes and presents combines test results and summary. By default, one process is used.

This option may not be combined with the --targets (-T) option.

--no-output

Do not produce a test results file.

-o file, --output file

Write full test results to file. Specify "-" (a hyphen) to write results to standard output. If neither this option nor --no-output is specified, the results are written to the file named results.qmr in the current directory.

-O file, --outcomes file

Treat file as a set of expected outcomes. The file must have be a results file created either by qmtest run, or by saving results in the graphical user interface. QMTest will expect the results of the current test run to match those specified in the file and will highlight differences from those results.

--random

Run the tests in a random order.

This option can be used to find hidden dependencies between tests in the testsuite. (You may not notice the dependencies if you always run the tests in the same order.)

--rerun file

Rerun only those tests that had unexpected outcomes.

The tests run are determined as follows. QMTest starts with all of the tests specified on the command line, or, if no tests are explicitly specified, all of the tests in the database. If no expectations file is specified (see the description of the --outcomes option), then all tests that passed in the results file indicated by the --rerun option are removed form the set of eligible tests. If an expectations file is specified, then the tests removed are tests whose outcome in the results file indicated by the --rerun option is the same as in the expectations file.

The --rerun provides a simple way of rerunning failing tests. If you run your tests and notice failures, you might try to fix those failing tests. Then, you can rerun the failing tests to see if you succeeded by using the --rerun option.

--result-stream descriptor

Specify an additional output result stream. The descriptor is in the format described in Section 3.4.2.

--seed integer

If the --random is used, QMTest randomizes the order in which tests are run, subject to the constraints described QMTest documentation. By default, the random number generator is seeded using the system time.

For debugging purposes, it is sometimes necessary to obtain a reproducible sequence of tests. Use the --seed option to specify the seed for the random number generator.

Note that even with the same random number seed, if tests are run in parallel, scheduling uncertainty may still produce variation in the order in which tests are run.

-T file, --targets file

Use targets specified in target specification file file. If this option is not present, the QMTest/targets in the test database directory will be used. If that file is not present, the tests will be run in serial on the local machine.

Notes:

• Because Firebird test suite depends on context values, if a file named context doesn't exists in the current directory, you must specify the context file with -C option.
• Each invocation of the qmtest run command is a single test run, and produces a single set of test results and statistics. Specify as arguments the names of tests and test suites to run. Even if you specify a test more than once, either directly or by incorporation in a test suite, QMTest runs it only once.

  If you wish to run all tests in the test database, use the implicit test suite . (a single period; see QMTest documentation), or omit all IDs from the command line.

summarize command

The qmtest summarize displays information stored in a results file.

qmtest summarize [ option  ...] [test-name | suite-name...]

The summarize command accepts the following options:

-f format, --format format

For details about this option, see the description of the qmtest run command.

-O file, --outcomes file

For details about this option, see the description of the qmtest run command.

--result-stream descriptor

Specify an additional output result stream. The descriptor is in the format described in QMTest documentation.

Back to Firebird QA.