HP3000/HP 3000 FAQ

Last Updated: Thursday, February 01, 2007 02:38 PM

HP3000 Database Logging

13.7. HP3000 Database Logging

MPE has a built-in facility to log transaction data. This logging facility is utilized by TurboImage on the HP3000 to record transaction information and to facilitate reporting and recovery of database transactions.

13.7.1. What can it do?

Image logging can be used in creating reports of database transactions or in the recovery of database transactions. Though there is no built-in facility to generate "audit" reports from an Image log file, there are several third party packages which will do this.

The ability to recover transactions from an Image log file on the other hand is "built in" to TurboImage.

The transaction manager is another facility built into MPE/iX that assists in maintaining the consistency of TurboImage databases.

13.7.1.1. Roll Forward vs. Roll Backward

The basic procedural difference is:

Roll forward recovery:

• A database is restored from a set of backup tapes
• Transactions from the log file (from the date/time the backup was taken until the present time) are applied to the database

Roll Backward recovery:

• Transactions from a log file are "undone" (i.e. backed out) from a live database until a given "state" or point in time is reached

Each method has it's advantages and disadvantages.

For instance, a database which is structurally "damaged" would likely need to be restored from a backup media, and transactions applied to it to "bring it up to date" (roll forward).

On the other hand, a database which is structurally intact but has been "damaged" by erroneous processing (perhaps someone accidentally deleted all of the week's orders) could use the log files to "roll back" the database, undoing the transaction(s) that corrupted the database, or back to a point in time before the error occurred.

13.7.1.2. Choosing logging to tape or disc

Image logging (actually the MPE logging facility which Image logging uses) allows you to direct transaction log files to disc or to tape devices.

13.7.2. What are the Limitations of Database Logging?

At present the Image logging facility is a CM (compatibility mode) subsystem and utilizes some CM data structures, which impose some performance penalties and a limitation on the number of concurrent processes that can access an Image database which is being logged. See below for details.

13.7.2.1. Performance Implications

Some estimates of the performance impact for databases which have logging enabled run as high as a 10% throughput reduction.

13.7.2.2. Concurrent Assessors

Due to limits in MPE's internal tables, the maximum number of processes that can share a log process -- which in this case means the maximum number of processes that can open a database with modify-access -- is 1140. Beware of this on limit on large systems.

13.7.3. How do I set it up?

1) You must setup a system logging process (see :GETLOG command) where you'll specify the log process "identifier", password, and the fileset (or tapeset) you'll have it writing to. You'll also specify whether you want full log files to "roll over" to new (sequentially numbered) files automatically (which you will if you're using DISC files.)

2) You'll need to start the log process (:LOG name,START). This is also a good time to add the appropriate commands to your system startup procedures to make sure the log process(es) get restarted when you restart the system (this must be done explicitly each time the system is restarted). It should also (preferably) be stopped gracefully before a system shutdown.

3) You'll need to associate the database with a logid (in DBUTIL >>SET base LOGID=logidname )

4) You'll need to enable the database for logging (again in DBUTIL >>ENABLE base FOR LOGGING)

5) Perhaps before (4), you'll want to ensure that you have modified your system backup procedures appropriately to make sure the appropriate log files get backed up. See the next FAQ item.

13.7.3.1. Integrating Logging with System Backups

At a minimum, for logging processes which log to disc, you'll likely want to do a :CHANGELOG command against each logging process when you start a system backup. This causes the log process to close the current file and open a new file; allowing you to ensure you have the maximum amount of recoverable data (right up to the point of the start of the backup) on your backup media. Without this, the current log file (which may have tens of thousands of transactions) will not get backed up because it is constantly "in use" by the logging process.

For users with "online" backup packages, a :CHANGELOG is probably sufficient (though consult the documentation with your backup product for their recommendations). This way your users can continue to access the database(s) while the backup runs.

For sites without online backups, you may actually want to STOP the various logging processes when you are about to start the backup (**Note that the :LOG xxx,STOP command, though it returns immediately, may take several seconds to complete, so factor that delay into your backup job stream/script). Once your backup completes, you can :LOG xxx,RESTART the log processes. A side effect of this; if the database is enabled for logging but the log process is not running, users/applications will not be able to open the database -- they get an Image error indicating the log process is not running. This can prevent someone from "sneaking in" and opening a database before the backup gets a chance to lock the file -- which causes your database to not get backed up.

13.7.4.  Getting Database activity reports from logging data

Image logging records every change made to the logged database(s). Every database accessor that opens the database in a mode capable of modifying data is recorded in the log files, with information including the Job/Session number, logical device, logon info, program used, and date/time. Database information logged includes records added (with complete record image), records deleted (again with a copy of the entire record that was deleted), and changes (which records the key information, followed by the changed fields with old and new values of each).

There are various software packages that can read these log files and produce very comprehensive records of database modifications.

Note for users attempting to write their own programs; the Image log file records are variable length record format, with the first 11 or so words being in a format documented under the "WRITELOG" Intrinsic (which Image uses to record the data); the remaining data in each record after that point is in the format as documented in the Image manuals. Accessing the data is somewhat tricky as the Image portion of each log record contains numerous "pointers" to fields farther along in the record. Also, Image fields are recorded by item numbers, so your program will need to have access to the real database and use DBINFO to get the actual set/item names (or you'll have to hard-code a list).