# eXtremeDB Transaction Logging Applications

> C/C++ applications using eXtremeDB Transaction Logging must first initialize the TL subsystem by calling `mco_translog_init()` before the database is opened via `mco_db_open_dev()`. In java and C# applications, the Database constructor initializes the TL subsystem when the database is instantiated.

> After transaction logging is initialized, C/C++ applications connect to the database as in any eXtremeDB application by calling `mco_db_connect()` or `mco_db_connect_ctx()`, Java and C# applications simply instantiate a Connection.

> At this point a save-point should be created by calling the `mco_db_save()` C/C++ function, (in Java and C# applications the `Connection.SaveSnapshot()` method) before logging is started. This establishes the starting point for all logged transactions to follow.

> Java and C# applications can recover a database from a saved image and log file as illustrated in the following code fragments extracted from \samples\csharp\tl\TLogBasicCS:

>             const string TL_LOG_FILE = "tlogbasic_tl_cs.log";
>             const String DBIMAGE_FILE = "tlogbasicdb_cs.bak";
>             Database.Parameters parameters = new Database.Parameters();

>             if (File.Exists(DBIMAGE_FILE))
>             {
>                 // set database image filename
>                 parameters.DatabaseSnapshotFilePath = DBIMAGE_FILE;
>                 loadedSnapshot = true;
>             }

>             // create Database object
>             db = new Database(new ExtremedbWrapper(),
>             Database.Mode.TransactionLoggingSupport);

>             db.Open("tlogbasicdb", parameters, devs); // open database.
>             if (loadedSnapshot)
>             {
>                 // Apply stored log file to databas
>                 LogReader logr = new LogReader(con, TL_LOG_FILE);
>                 LogReader.LogInfo linfo = new LogReader.LogInfo();
>                 logr.Apply();
>             }
>             con.SaveSnapshot(DBIMAGE_FILE);
>         }

> In C/C++ applications, the following additional log file operations can be performed on log files:

> > A log file may be marked with a label at any point by calling the function `mco_translog_label()`, which provides the ability to restore the log up to that label. Also, a log file may be truncated and restarted by the function `mco_translog_truncate()`. Note that it is wise to create a database save-point prior to truncating the log file, otherwise the transactions will be unrecoverable. Further, it is wise to create the save-point to a different (new) file than the previous save-point. If you overwrite the existing save-point and the system crashes prior to completing the new save-point, there can be no recovery. So, for safety, the sequence should be:

> > > 1. create a new save-point

> > > 2. truncate the log file

> > > 3. delete the previous save-point file

> > If transaction logging is not synchronous (see `MCO_TRANSLOG_SYNC_INSTANTLY` below), a log file may be flushed to disk at any time by calling the function `mco_translog_flush()` to force immediate flushing of the file system buffers related to a log file. However, automatic flushing strategies (activated by elapsed time or by transaction count) can be specified when starting transaction logging (see the function `mco_translog_start()`) that eliminate the need for an application to explicitly flush the log file.

> > Information about the current log file (the current log size, the count of stored transactions and other information) may be gathered during the logging process by calling the function `mco_translog_get_info()`. Or, information can be obtained about a specific log file on persistent media (the ability to apply or append to this log, the log file size and other information) by calling `mco_translog_query_info()`.

> > A database can be restored from compatible log files by calling the function `mco_translog_apply()`. Note that a proper save-point, i.e. a previously stored database image, should be restored via `mco_db_load()`, or a new empty database should be opened before applying a log file(s). Then, `mco_translog_apply()` is called to replay any transactions committed after that database image was saved. Before applying the log(s) to the database, the TL runtime first checks for dictionary compatibility, runtime modes and data size compatibility, and finally verifies that the current value of the database transaction count is the same as that stored in the log. If all is in order, `mco_translog_apply()` proceeds to apply all transactions stored in the log file, after which `mco_translog_start()`is called to begin transaction logging for the new database session.

> ## Database Recovery

> The sequence to recover a database from a save-point or an empty database as the starting point, and two or more consecutive log files is shown here:

> Transaction Logging:

>         char buff[64];
>         mco_TL_start_data_t log_parms;
>         log_parms.flags = 0;
>         sprintf(buff, "transactions_%d.log", 0);
>         CHECK(mco_translog_start (db, buff, &log_parms));
>         log_parms.flags |= MCO_TRANSLOG_RESTART;

>         for (i = 1; i < NPARTS; i++)
>         {
>             /* do database activity here */
>             use_database();
>             /* under some condition, e.g. log file size, restart log with new file */
>             if( /* some condition */ )
>             {
>                 sprintf(buff, "transactions_%d.log", i);
>                 CHECK(mco_translog_start (db, buff, &log_parms));
>             }
>         }
>         CHECK(mco_translog_stop(db));

> Database Recovery:

>         for (i = 0; i < NPARTS; i++)
>         {
>             char buff[64];
>             mco_TL_log_info_t info;

>             /* load next log file */
>             sprintf(buff, "transactions_%d.log", i);
>             CHECK(mco_translog_query_info(db, buff, 0, 0, &info));
>             ASSERT(info.transaction_apply_compat, MCO_YES);
>             CHECK(mco_translog_apply(db, buff, MCO_TRANSLOG_ALL_LABELS));
>         }

> This sequence fully recovers the database. In the normal course of events, the database is restored from the last save-point image that was saved during an orderly shutdown, and there is no transaction log(s). In the case of an abnormal termination, transactions from the log file will be recovered but, because the application terminated without calling `mco_translog_stop()`, there is no proper end of file mark at the end of the (last) log file and `mco_translog_apply()` will return `MCO_S_TL_INVDATA`. In this case, you can call `mco_translog_query_info()` (which will also return `MCO_S_TL_INVDATA`) and the count of transactions actually restored by `mco_translog_apply()` will be returned in `info.stored_trans_count`. (Whether the last restored transaction is the last transaction actually performed before the application crashed will depend on the flushing policy employed, and, if any policy other than synchronous logging, then also luck).

> When the application has finished, transaction logging is halted by calling `mco_translog_stop()`. Then normal application termination is affected by calling `mco_db_disconnect()` and `mco_db_close()`. The following pseudo-code snippet illustrates how an application might perform a “restore and proceed” scenario (see samples/tl/tlogbasic for a complete implementation:

>         mco_runtime_start();
>         mco_translog_init();
>         ...
>         mco_db_load();
>         mco_db_connect();
>         mco_translog_query_info(&info);
>         if (info.stored_transactions > 0)
>         {
>             mco_translog_apply();

>             /* Create initial save-point */
>             mco_db_save();
>         }

>         mco_translog_start();
>         while (run) /* Normal database usage */
>         {
>             ...
>             if (some_condition ) /* periodically create save-point */
>             {
>                 mco_db_save();

>                 /* Truncate the log file */
>                 mco_translog_truncate();
>             }
>         }
>         mco_db_save(); /* create the last save-point */
>         mco_translog_stop();

>         /* delete the log file here; it’s no longer needed */
>         mco_db_disconnect();

> ## Tips

> There are three cases when using TL is warranted:

> > 1. When the database consists of only transient (in-memory) objects or when it consists of both transient and persistent objects (a so-called ‘hybrid’ database) and you need durability for both the persistent and the transient data.

> > 2. To export transactions to an external system (e.g. another DBMS), which we refer to as “Data Relay”.

> > 3. To create a persistent queue of events, for more robust event handling than is possible through the ‘event’ DDL notation and associated synchronous and asynchronous event handlers.

> Do not use TL to add persistence for persistent-only databases; eXtremeDB runtime has its own robust and efficient database recovery facilities for persistent databases. It is important not to confuse the use of eXtremeDB Transaction Logging APIs with the normal eXtremeDB runtime logging for persistent databases.

> Due to the fact that applying a log to a database is slower than loading the database from a save-point (via function `mco_db_load()`), you should use TL function `mco_translog_apply()` only in case of an application crash or if a log needs to be applied up to specific label. So create database save-points regularly, then when the application finishes normally, a ‘restore and proceed’ scenario like that shown above can be applied.

> ## DDL Requirements

> The schema for a database that uses transaction logging must include the `auto_oid` declaration.

>         declare auto_oid [ESTIMATED_NUMBER_OF_OBJECTS];

> Whether the database actually uses transaction logging is determined at run-time. When the data definition is compiled for transaction logging, the eXtremeDB DDL compiler inserts an 8-byte unique identifier (called `auto_OID`, not to be confused with the field type `autoid`) into each object.