# Incremental Backup Implementation

> The incremental backup process creates a backup file which stores database pages in blocks marked by a binary session header and footer structure (called a backup record). A single backup file can contain several independent backups, possibly of different types (full or incremental) or with different tags or timestamps.

> For all-in-memory databases, only the backup file is created. For persistent databases, the database close operation causes an additional Backup Map file to be created.

> ## Backup Map

> This map file contains information used to determine the pages that were modified between one incremental backup and another. For a persistent database, the eXtremeDB runtime maintains counters (the backup bitmap). When the application shuts the database down, the state of the counters needs to be saved and then restored when the application opens the database again.

> Without this map file a full database snapshot would be needed after every application shutdown and restart since the backup procedure would have no data about modifications made to the database pages. So, when an application opens a persistent database with incremental backup enabled, the eXtremeDB runtime will create the backup bitmap file on every database shutdown, whether the backup API is actually called or not.

> The backup map keeps track of database pages processed by the backup algorithm. To perform an incremental backup, the runtime needs to track modifications of all the database pages. This is done by using integer counters to indicate the pages' content versions. Every time a database page is modified the runtime increases a counter associated with this page.

> It would be untenable to keep a separate counter for each database page since a database could require several terabytes of storage. So the runtime instead keeps a counter for a cluster of pages. The size of that cluster depends on the parameter `backup_map_size` which determines the total number of counters for the database. The runtime calculates how many counters can fit in the specified `backup_map_size` and then how many pages will be served by a counter (a cluster). Consequently, the number of pages in a cluster is less for a larger map which means that the backup procedure will store fewer pages for every modified page to the backup file. Therefore, the backup process will be faster and the backup file will be smaller. On the other hand, the map is allocated from the database memory buffer, so the buffer needs to be bigger for bigger maps.

> The page modification tracking mechanism is similar for in-memory and persistent databases (and for hybrid databases the in-memory part and the persistent part of a database). The difference is the location of the page version counters: for a persistent database, the counters reside in the backup bitmap and for an in-memory database, the counters are contained in the data pages. The size of the memory allocated for the database is known from the database open call, so there is no need for the `backup_map_size` parameter for the in-memory part as the runtime calculates this automatically. But for persistent databases, a reasonable size should be specified.

> A reasonable map size can be calculated as follows:

>         sizeof(int) * <expected-persistent-part-size-in-bytes> / <disk-page-size> / <disk-pages-per-backup-cluster>

> where `disk-pages-per-backup-cluster` is the number of pages that will be stored into the backup map if even a single page of that cluster is modified. For example, suppose that the `disk-page-size` is 4K and we would like to store modifications in bunches of 64K; then `disk-pages-per-backup-cluster` would be `64 / 4 = 16`. Then to calculate `backup_map_size` for a 1G database on a 64 bit system (sizeof(int) = 4), the math would be as follows:

>         4 * 1G / 4K / 16 = (4G / 4K) / 16 = 1M / 16 = 64K

> If no `backup_map_size` is specified, then the eXtremeDB runtime calculates it using the maximum persistent database size, which is `MCO_INFINITE_DATABASE_SIZE` (1 terabyte for 64-bit databases and 4G for 32-bit databases); so the default `backup_map_size` is calculated as:

>         64bit: 4 * 1T / 4K / 16 = (4T / 4K) / 16 = 1G / 16 = 64M

>         32bit: 4 * 4G / 4K / 16 = (16G / 4K) / 16 = 4M / 16 = 256K

> ## Backup Process

> The backup is an iterative process controlled by the two conditions `backup_max_passes` and `backup_min_pages`. The procedure periodically scans the database for modifications while it works in parallel with other application activities that may interact with the database runtime. On each pass it counts the number of modified pages. When it reaches the threshold determined by parameter `backup_min_pages` or `backup_max_passes`, the final exclusive backup pass locks the database for the final pass when it writes modified pages to the backup file. The parameter `backup_max_passes` defines a limit on the number of backup procedure scans of the database. If the database usage is such that so few pages are modified that the `backup_min_pages` threshold may not be reached, `backup_max_passes` can be set to a small number of passes so that when it is reached the runtime will block database operations and initiate the final exclusive backup pass.