Unloading and Loading a ClassiX^® database

1. Objective and Course of Action

2. Program Fetch

2.1 Program fetch parameter CXUOSR.EXE (unload)

2.2 Fetch parameter for CXCOSR.EXE (generating checksum)

2.3 Program fetch parameter CXLOSR.EXE (load)

2.4 Example – a batch file

3. Clustering

4. Database analysis

5. Object cross-reference

1. Objective and Course of Action

When unloading the database, it gets written in the ASCII format (DATABASE.DMP, FREEOBJ1.DMP, ...) in sequential files. ASCII is technically also readable for people. The files contain all information about database objects that are required to rebuild these objects in a new database with the same data and to reconstruct all relations between the objects. Doing this, object division into sub-databases, segments and clusters can be modified, which is usually the reason for such operations.

Unload - Load can be used instead of the schema evolution which is provided by ObjectStore (database migration to a new schema - modified class definitions). Unloading, the following information will get lost:

• objects, which cannot be accessed (neither via root entry point collection, nor via relation from at least one other accessible object)
• indices via collections (including index manager)
  

Logically deleted objects will persist, as long as they are referenced directly or indirectly by an active object.

The total procedure - including checksum creation to assure logical integrity - involves four sub-steps:

• Unload with program CXUOSR.EXE
  In phase 1, all objects are unloaded in the root entry point collections. In phase 2, for every object that has been unloaded so far it will be determined,  if its relations reference objects, that have not yet been unloaded and which will then be unloaded. Since such objects also undergo this test, the method is transitive and at the end of the phase all accessible objects will be unloaded. This requires a hash table with the database addresses of all objects, that have been unloaded so far (TMPFILE.DMP). In phase 1 unloaded objects are in the file DATABASE.dmp, phase 2 objects in the files FREEOBJ1.dmp, FREEOBJ2.dmp, ... Per object, another file is created, that contains the object data stream. 
  Phase 3 is optional: the object allocation is rescued in the root entry point collections of file REP_COLLS.dmp.
  
• "Old database" checksums build with CXCOSR.EXE (Parameter –1)
  All database objects are read segment by segment and evaluated with a checksum. Reading the segments, even not accessible objects can be "seen". These objects and accessible objects can be told apart with the hash table, which has been created while unloading.
  
• Loading the new database with CXLOSR.EXE                                                                                                                                          Database reconstruction also splits up into two phases. In phase 1, all objects are generated and the according values are assigned to the data fields. Phase 2 is for reconstructing object interrelations. To make this possible, a hash table with all object database addresses is generated in phase 1. This is similar to unload, but here every entry also contains the new database address: this way, "old database" references can be translated into "new database" reference. It is possible to control the ordering process of new objects into the root entry point collections via CLASSIX.INI file - and therefore also to modify it in relations to the unloaded database - or to recreate it identically via information stored in phase 3.
• "New database" checksum with CXCOSR.EXE (parameter –2)                                                                                                                     A checksum gets calculated for every object in the newly loaded database. A match with the "old database" checksum guarantees for a correct object data field recreation. Furthermore, the number of relations from every object gets evaluated, which also controls the object relation recreation. Whenever there are any occurring discrepancies, it is possible to view the according objects in both databases.

This is the procedure, if phase 3 of the unload- and load program is not used. The object order in the root entry point collections is determined via information in CLASSIX.INI. The explicit object pasting in phase 3 guarantees an absolutely identical object assignment to the root entry point collections (see Procedure).

Information: The newly built database can only be unloaded instead of the unloaded "old" database, if the total checksums in CHECKSUM.OLD and CHECKSUM.NEW are identical

2. Program Call

All flags can start with / or -. Capital letters and lower-case letters are equal. A parameter coming after a flag can follow directly or be separated via blanks; e.g.: 

cxuosr /iTS.INI /h5000000

or

cxuosr –I ts.ini /H 5000000

If successfully, the programs provide the return code 0.

The two phases

• unloading root entry point collection objects
• unloading objects, which could only be accessed via objects that have been unloaded so far 

are executed automatically one after the other, if neither /1 nor/2 has been indicated. While unloading, it is specified how the objects are clustered in the new database. Phase 3 collects information about the object order in the root entry point collections.

Phase 1 is restart-capable. Condition: hash table is maintained storage resident (parameter /E or /F).
If a class had been unloaded completely after the predetermined time had run out, the hash table is saved as file 'filename.nnn', with nnn = { 001, 002, ...    }and the unload file is labeled (Checkpoint).
From here, it is possible to continue: Parameter /R. The generated unload data parts have to be joined in the end, eliminating redundant information - call with parameter /P. The following overview shows all other flags that are relevant for restart-capability:

The file name of the saved hash files (complete path) has to be indicated as 'restartInfo' (see Example).

2.2 Fetch parameter for CXCOSR.EXE (generating checksum)

For each single object, /1 makes the checksums get written into the file OBJECTS.OLD. If CXCOSR.EXE runs for the new database via /2, the file OBJECTS.NEW forms, which holds all objects including checksums. Characteristic for checksum differences is, that it is possible to view the relevant objects via InstantView^® in the new database, as well as in the old one. E.g. with the object inspector – InstantView^® source: CHECKSUM.CXP. The  CX_DB_UTILITY object provides further functions for object analysis in the old and new database.

The program CXCOSR.EXE provides another function, which doesn't directly belong to database loading and -unloading: the construction of an Object Cross-Reference – a help database, which can define from which other objects an object gets referenced.

2.3 Program Fetch Parameter CXLOSR.EXE (loading)

Usually (neither /1 nor /2 indicated), both phases are automatically executed one after the other. CXLOSR.EXE generates a protocol in the LOAD.LOG file.

The hash table (GATEWAY.DMP) forms the bridge between old and new database: The according object location in the new database is known for each object Location in the old database. The table can be kept resident in the memory while loading: local (flag/F) or get distributed across external servers (flag /E). A "distributed gateway file" can be pooled in one file.

2.4 Example – a Batch File

The ObjectStore data transfer between the database server and the client is page-oriented. Therefore the database object order has great influence on the application performance.

Object clustering can be controlled for

• object creation with InstantView^® commands CreatePersObject, CopyPersObject
• object creation when unloading/loading a database.

The clustering can be activated/deactivated step by step in both cases.

For the clustering, an object is either a master- or a slave object:

A master object gets saved in a database segment, that has been determined for its class; a cluster might be created for the master object. Segment information and information about the cluster are described in the database layout (file CLASSIX.ODB).

After deactivating the clustering completely, there are only master objects - this is the default option in ClassiX^®.

The slave object position in a database determines another database object, which it gets connected to. Generating such an object with CreatePersObject, CopyPersObject needs to be delayed until a relation to an already existing database gets built. Therefore, the named    InstantView^® commands generate a (transient) CX_LAZY_CREATOR object class.
The unload/load procedure is explained below.

First, this requires to determine the rules to define, whether an object is master or not.

The clustering rules have been split up into categories. Assigning a bit to each category, creates a pattern, which lets the categories get switched on and off independently.

This pattern is set with the environment variable CX_CLUSTERING; default option is CX_CLUSTERING=0.

The following table shows the categories and the according rules; column 1 lists the bit and column 2 the according name.

Column 3 indicates which class can be a slave object. While the class is the only necessary criterion for basic objects (e.g. the basic classes CX_DATE, CX_VALUE ...), there are more conditions for more complex objects:

• A... means class A and all its derived classes
  
• REP_pattern is the bit pattern, defining the object order inside the REP collection; either predetermined via class description in CLASSIX.ODB or via explicit CreatePersObject indication
  
• Conditions that refer to the master object "master" only apply while loading/unloading, a CX_LAZY_CREATOR is always created for CreatePersObject / CopyPersObject
  
• If an object gets created with CreatePersObject, the additional condition REP_pattern =0 applies for all categories. A transient object CX_LAZY_CREATOR cannot be inserted into a REP collection.
  

The rules above should not only control the clustering, which has been generated via delayed object creation (CreateOersObject -> CX_LAZY_CREATOR). Moreover, it should also be active when unloading/loading a database.

New database object order is already determined when unloading:

Unloading a master object o, all objects  o_r that are directly accessible from here via relations (Pointer/Collections) are checked for the rules above. Here, the objects o take over the master role. If there are slave object among the objects o_r, they are unloaded next and marked in the unload file as objects belonging to o.

When unloading the slave objects, the same happens - o_r takes over the master role and all accessible objects are tested, whether they can be "clustered" by o_r. These indirectly accessed objects also go into the cluster belonging to the object o. When loading the database, object o defines, how to save all its dependent objects. 

A just unloaded object o "asks" all its referenced objects, whether they can be "clustered" by o.

According to this, it is important, in which order the objects are unloaded.

When unloading, the priority numbers (0 to 5) are assigned to the classes. In phase 1, all objects are unloaded out of the root entry point collections, whereas classes with priority numbers come first:

Within one category, classes are unloaded according to order shown in the class description in CLASSIX.ODB, meaning
the class statement order in CLASSIX.ODB influences the clustering.

The database needs have only one corrupt object to make the unload procedure get cancelled with a bug.

Program CXAOSR.EXE can find such errors in an analysis run and write them into a report file:

Example: Analysis runs for a large database spread across three PCs: cxaosr /A-33%, cxaosr /A34%-66%, cxaosr /A67%-

Information: Don't forget parameter syntax in a batch file: double percentage character: e.g.: specify cxaosr /A34%%-66%%!

Analyzing (with Flag /A...), the virtual function SanityCheck() gets called for every object. A return code > 0 signalizes an error:

Objects

SanityCheck calls itself also for DDI members, as long as it is an object. In case of an error, original error code + 10000 gets returned.

Furthermore, relations to other objects are analyzed (function CheckReferences):

CX_DB_UTILITY object functions support an effective error log analysis.

Sometimes it would be handy to know all referenced objects o_r  for one object o, all objects that have a pointer to o or a collection with o as an element.

This information is kept in CX_XREF objects. Building a Cross-Reference means, that exactly one CX_XREF object gets generated for each object, which then points at all objects referencing the object that is represented by the CX_XREF object.

It makes sense, to keep the CX_XREF objects in a separate physical database - since they just represent a snapshot which will be outdated soon. CX_XREF objects refer to "normal" objects - but not the other way around. Therefore, the physical sub-database for the cross-reference can be deleted at any time, without an effect on the "actual" database.

An objects assignment to "its own" CX_XREF object is realized  outside of the database (in the  gateway file).
The object manager function GetObjectsReferringTo() provides a collection of all objects o_r referencing this object o for an object o, which has been passed as a parameter.

No corresponding CX_XREF object gets generated for such an object o, which no other object o_r references to, unless it is required, that a CX_XREF object is generated via flag /O for every object to find single object in the database.

Describing a help database for the cross-reference:

Working with the cross-reference, this description is added to the Initialization File (e.g. classix.ini).

Information:
Due to performance, the CX_XREF object REP Collection should be of the type LIST.
A corresponding CX_XREF object is generated for every source database object, which is referenced by another one (which is the case for most objects).  For a larger number of objects, the integration into a SET will take longer. For large amounts of data, there should be a limitation for the Segment Splitting. CX_SLOT_XREF objects are generated in the help database for dynamic database cross-reference. Meta information for object- and slot cross-reference are shown in the following example.

The object cross-reference gets built by generating the help object sub-database with 

    ...

    SET CX_XREFDB=myXReferenceDB.cxd

    cxgosr –U –I modifiziertesIniFile.ini –batch

After, the cross-reference is built via CXXOSR.EXE:

The functions GetObjectsReferringTo() require gateway files. 
Gateway files (parameter /E) spread across several external DCOM servers and can therefore be pooled with program CXXOSR.EXE:

The correct Teilgrößen specification (n₁ + n₂ + ... = n_total) is compared to the gatewayFile_kspecifications.  

Here you can view example batch files to generate the complete cross-reference.

Partial cross-reference:
The creation of an object cross-reference can be sped up, whenever it only needs to be found out for a certain amount of objects { o₁, o₂, ... }, which objects o_r  reference one of the given objects { o₁, o₂, ... }. Parameter /X indicates a file, which

• contains the estimated number of objects n and the identifier EXCLUDE in the first line
• defines the following objects in the database with their location

The amount n is used to calculate the hash file prime number - a larger n causes better performance. EXCLUDE requires the specified objects not not to be tested as potential referents. This makes sense when  repairing corrupt objects in the database: the objective is to find out, which objects refer to "corrupt" objects, while excluding these corrupt objects from the test.

Example:

2000, EXCLUDE
<0|284|0|80f230|10000>, <0|284|0|80f350|10000>, <0|284|0|80f470|10000>, <0|284|0|80f590|10000>, <0|284|0|80f7e0|10000>, <0|284|0|80fa20|10000>, <0|284|0|80fb40|10000>, <0|284|0|80fc60|10000>, <0|284|0|80fd80|10000>, <0|284|0|810360|10000>, <0|284|0|8105bc|10000>, <0|284|0|8106ec|10000>, <0|284|0|810800|10000>, <0|284|0|810920|10000>, <0|284|0|810a40|10000>, <0|284|0|810b60|10000>, <0|284|0|810c80|10000>, <0|284|0|810da0|10000>, <0|284|0|810ec0|10000>, <0|284|0|81157c|10000>,
<0|284|0|811914|10000>, <0|284|0|811a84|10000>, <0|284|0|812040|10000>, <0|284|0|812268|10000>, <0|284|0|812320|10000>, <0|504|0|7d13cc|10000>

Comma as a separator between elements is optional. The number of objects per line is also optional, as long as it is less than 512 characters. 

Hide certain objects:
Even for a complete cross-reference, it is possible to exclude single objects. This method was meant to help when dealing with a corrupt database. Display file structure as shown above.

Database analysis and partial cross-reference form a tool to "repair" logically or physically corrupt/inconsistent objects in a database (Example procedure).

Information: An object can be replaced by another one via cross reference (function substitute, DB utility object).