Firebird Database Housekeeping Utility

The command to activate a database shadow is:

This makes the shadow file the new database file, and the users are able to continue processing data as normal and without loss.

In the event that your main database file(s) become corrupted or unreadable, the DBA can activate a shadow file. Once activated, the file is no longer a shadow file, and a new one should be created to replace it. Additionally, the shadow file should be renamed (at the operating system prompt) to the name of the old database file that it replaces.

Once a shadow file has been activated, you can see the fact that there are active shadows in the output from gstat:

The command to kill all unavailable database shadows, for a specific database, is:

In the event that a database running with shadow files loses a shadow, or a shadow becomes unusable for some reason, the database will stop accepting new connections until such time as the DBA kills the faulty shadow and, ideally, creates a new shadow to replace the broken one.

The following (contrived) example, shows what happens when the database loses a shadow file, and an attempt is made to connect to that database. There are two sessions in the following example, one is connected to the database while the second deletes a shadow file and then tries to connect to the database. The command line prompts shows which of the two sessions we are using at the time.

First, the initial session is connected to the database and can see that there are two shadow files attached:

In the second session, we delete one of the shadow files, and then try to connect to the database

The second session cannot connect to the database until the problem is fixed. The DBA would use the gfix -⁠k[ill_shadow] command to remove details of the problem shadow file from the database and once completed, the second (and subsequent) sessions would be able to connect.

The database now has a single shadow file where before it had two. It is noted, however, that gstat still shows the database as having two shadows, even when one has been removed.

The gfix command -⁠l[ist] will display details of transactions that are in limbo. If there is no output, then there are no transactions in limbo and no further work need be done. The command is:

An example of listing limbo transactions is shown below. This command is run against the local database on the server named linux where a multi-database transaction had been run connected to databases linux@my_employee and remote:testlimbo. Both of these database names are aliases.

If the command is run against the remote database then nothing will be listed because that database does not have any limbo transactions — the transaction that went into limbo, when the network failed, for example, was initiated on the local database.

You may also supply the -⁠pr[ompt] option to the command, and you will be prompted to COMMIT or ROLLBACK each detected limbo transaction. In this case, the command would be:

An example of this is shown below.

When a limbo transaction has been detected, the DBA has the option of committing or rolling back one or more of the transactions reported as being in limbo.

When more than one transaction is listed, the DBA can either commit or roll back all transactions in limbo, or a specific transaction number.

The following commands show the -⁠co[mmit] option being used, but the -⁠r[ollback] option applies as well, it all depends on what the DBA is trying to achieve.

To commit every limbo transaction on the database, the following command would be used:

If the DBA wanted to commit a single transaction, then the command would change to the following:

Where TXN is the transaction number to be committed.

When either of these options are used, there is no feedback from gfix to advise you that the commit actually worked. You would need to rerun the gfix -⁠list command to make sure that all, or the selected, limbo transactions had indeed gone.

You cannot commit or rollback a transaction that is not in limbo. If you try, the following will occur:

When committing or rolling back all limbo transactions, the -⁠pr[ompt] option can be specified. It is, however, not permitted when processing a single transaction. An example of using the -⁠pr[ompt] option has been shown above under listing limbo transactions.

Gfix can be used to perform automatic two-phase recovery. The command for this is -⁠tw[o_phase] and, like -⁠co[mmit] and -⁠r[ollback] above, requires either all or a transaction number.

The output of the -⁠l[ist] command shows what will happen to each listed transaction in the event that the DBA runs the -⁠tw[o_phase] command.

The command also takes the -⁠pr[ompt] option, as above, when used to process all transaction.

The command line to carry out automatic two-phase recovery is:

or

As above, TXN is a single transaction number from the list of limbo transactions.

Garbage, for want of a better name, is the detritus that Firebird leaves around in the database after a rollback has been carried out. This is basically a copy of the row(s) from the table(s) that were being updated (or deleted) by the transaction prior to the rollback.

Almost all garbage is created by committed transactions. Since around V2.5 transactions that rollback are cleaned up immediately — assuming that Firebird is still running.

The major cause of garbage build-up is long-running transactions that require Firebird to keep old versions of records that are frequently updated. Another source of garbage is an application strategy that deletes records and never revisits them.

What actually happens on delete is that Firebird stores a "deleted stub" with the full record as a back version. Until the delete is mature — meaning that all active transactions started after the delete was committed — the old version must be preserved.

Imagine a table that’s indexed and accessed by date. On some schedule, records age out and are deleted. In the application, records are accessed by date, and the deleted records are so old no query ever asks for them. So there they sit, taking up space and doing no good to anyone. Even with a garbage collect thread, some active transaction has to stumble over deleted records or records with unneeded back versions before the record will be garbage collected.

In cooperative garbage collection, that particular record will be cleaned up immediately (or at least when the transaction gets some cycles). The dedicated garbage collection thread should clean up all the records on a page, but not until an active transaction tells it that there’s a page that needs cleaning.

Because Firebird uses multi-generational architecture, every time a row is updated or deleted, Firebird keeps a copy in the database. These copies use space in the pages and can remain in the database for some time, especially if there are no active transactions stumbling across them!

There are a number of causes of garbage:

With regard to the remnants from aborted or rolled back transactions, Firebird (now) carries out record-keeping to facilitate save points. This housekeeping allows Firebird to identify and, if necessary, undo all changes made by a transaction in the event that it is rolled back, or which failed due to a lost connection.

If a failed transaction is rolled back in either manner, its state is set to committed as there are no differences between a failed transaction and one in which it committed after making no changes.

These remnants are simply older copies of the rows that were being updated by the respective transactions. The differences are that:

Firebird will automatically sweep through the database and remove the remnants of rolled back transactions and this has two effects:

In the Super Server version of Firebird 2.0, garbage collection has been vastly improved. There are now three different ways of operation and these are configurable by setting the GCPOLICY parameter in the firebird.conf configuration file. By default, Super Server uses combined while Classic Server uses cooperative. The other option is background.

The default sweep interval for a new database is 20,000. The sweep interval is the difference between the Oldest Snapshot Transaction, or OST and the Oldest Interesting Transaction or OIT.

An interesting transaction is one which has not yet committed. It may be still active, in limbo or may have been rolled back. (Limbo transactions are never garbage collected.)

The sweep facility runs through the database and gets rid of old rows in tables that are out of date. This prevents the database from growing too big and helps reduce the time it takes to start a new transaction on the database.

You can check if a manual sweep may be required by running the gstat utility to check the database header page and extract the Oldest Transaction (OIT) and Oldest Snapshot (OST) numbers from the output. If OST - OIT is small (less than the sweep interval) then a manual sweep may be in order. The SHOW DATABASE command in isql will also show the details you need.

Alternatively, another idea is to run gstat with the switches set to show old record versions. If that shows a problem, then it may be a good idea to start looking for long running transactions.

The options for this are:

For example:

The information is shown in the 'record versions' statistic. In this example, there are 15 versions and as the 'total records' is also 15, then all the records have been deleted and need garbage collecting.

A manual sweep can be run by using the -⁠s[weep] command. (See below).

To alter the database’s automatic sweep interval, use the following command:

The INTERVAL parameter is the new value for the sweep interval. The database name parameter is the database upon which you wish to alter the setting for automatic sweeping. The following example shows the setting being changed from the default to a new value of 1,000.

If automatic sweeping has been turned off, or only runs rarely because of the sweep interval setting, the DBA can manually force a sweep to be performed. The command to carry out this task is:

This command will force the garbage left over from old rolled back transactions to be removed, reducing the database size and improving the performance of new transactions. Rolled back transactions are less of a problem than old versions from committed transactions, however, when the newest versions is being used by all current and future active transactions.

The -⁠i[gnore] option may be supplied. This forces Firebird to ignore checksum errors on database pages. This is not a good idea and should rarely need to be used, however, if your database has suffered some problems it might be necessary to use it.

The following example shows a manual database sweep being implemented:

If you set the sweep interval to zero then automatic sweeping will be disabled. This implies that there will be no automatic housekeeping done so your database performance will not suffer as a result of the processing requirements of the automatic sweep.

If you disable sweeping you are advised to run a manual sweep at regular intervals when the database is quiet. Alternatively, simply make sure you take regular backups of the database and as this is something you should be doing anyway, it shouldn’t be a problem.

If there is maintenance work required on a database, you may wish to close down that database under certain circumstances. This is different from stopping the Firebird server as the server may well be running other databases which you do not wish to affect.

The command to close a database is:

The TIMEOUT parameter is the time, in seconds, that the shutdown must complete in. If the command cannot complete in the specified time, the shutdown is aborted. There are various reasons why the shutdown may not complete in the given time, and these vary with the mode of the shutdown and are described below.

The OPTION parameter is one of the following:

When a database is closed, the SYSDBA or the database owner can still connect to perform maintenance operations or even query and update the database tables.

Once all maintenance work required on a database has been carried out, you need to restart the database to allow normal use again. (See shutdown option above for details of closing a database.)

The -⁠o[nline] command allows a database to be restarted. It takes a single parameter which is the database name as follows:

The following example shows a closed database being started.

The above discussion of stopping and starting a database apply to all versions of the server up to version 2.0. From 2.0 the commands will work as described above, but a new state has been added to define exactly how the database is to be stopped or started. The commands change from those described above to the following:

STATE is new in Firebird 2.0 and is one of the following:

There is a hierarchy of states for a database. The above list shows them in order with normal at the top and full at the bottom.

This hierarchy is important, you cannot shutdown a database to a higher or equal level that it currently is, nor can you startup a database to a lower or equal level.

If you need to identify which level a database is currently running at, gstat will supply the answers. The following example puts a database fully online then progressively shuts it down to fully offline. At each stage, gstat is run to extract the Attributes of the database.

Sometimes, databases get corrupted. Under certain circumstances, you are advised to validate the database to check for corruption. The times you would check are:

When a database is validated the following checks are made and corrected by default:

There are options to perform further, more intensive, validation and these are discussed below.

If the database validation described above produces no output then the database structures can be assumed to be valid. However, in the event that errors are reported, you may have to repair the database before it can be used again.

The gstat seems to take some time to respond to the addition of shadow files to a database. After adding two shadows to a test database, gstat still showed that there was a Shadow count of zero.

Even worse, after killing the second shadow file and running the DROP SHADOW command in isql to remove the one remaining shadow file, gstat decided that there were now three shadow files in use.

Even using Firebird version 2 it appears that many commands, which fail to complete without an error, return a response of 0 to the operating system.

For example, the following shows two attempts to shut down the same database, the second one should fail — it displays an error message — but still returns a zero response to the operating system. This makes it impossible to built correctly error trapped database shutdown scripts as you can never tell whether it actually worked or not.

Under classic server, using the -⁠fo[rce_shutdown] option to the -⁠sh[utdown] command acts exactly the same as the -⁠at[tach] option.

There are a couple of problems with limbo transactions as discovered by Paul in his testing.