pgl-backup (PixelGardenLabs Backup) is a simple, powerful, robust, high-performance and cross-platform file backup utility.

It is designed for creating versioned backups of local directories to another local or network-attached drive. It supports periodic snapshots, an efficient incremental mode with a flexible retention policy, and automatic compression. A core design goal is ensuring backups can be restored no matter what; by relying on standard file structures and open archive formats, your data remains fully accessible via native OS tools without needing pgl-backup installed.

Based on a philosophy of "solidity over frills," pgl-backup is built for users who value reliability and performance above all else.

• Engineered for Robustness: The backup engine uses a "Stage, then Process" pipeline with atomic operations to ensure your backup repository is never left in a corrupt or inconsistent state, even if a run is interrupted. Critical steps are "fail-fast," while non-critical failures (like a single file error) are logged without halting the entire backup.

• High Performance by Default: A concurrent, producer-consumer architecture maximizes I/O throughput. It includes specific optimizations like a sequential-write mode for HDDs and memory pooling to reduce GC pressure, ensuring fast performance on both modern SSDs and traditional spinning disks.

• Zero Vendor Lock-In: Your data is your own. Backups are stored in standard folders and open archive formats (.zip, .tar.gz, .tar.zst). You can always access and restore your files using native OS tools, with or without pgl-backup.

• Safety First: Extensive pre-flight checks prevent common pitfalls before they happen. This includes "Ghost Directory" protection (for unmounted drives), cross-platform case-sensitivity mismatch detection, and path nesting validation to prevent infinite loops.

• Simple and Transparent: A single, dependency-free binary that's easy to deploy and automate. The configuration is straightforward, and the logging is clear, giving you full visibility into what's happening.

• Core Concepts
• Key Features
• Architecture & Cloud Sync Safety
  • Why This Matters for Cloud Sync
  • Which Directories to Sync?
• Installation
  • From Source
  • From Pre-compiled Binaries
• Security and Binary Verification
  • Running on macOS (Gatekeeper)
  • Running on Windows (SmartScreen)
  • Verifying Checksums
• Getting Started: A 3-Step Guide
  • Step 1: Initialize or Update Configuration
  • Step 2: Review Configuration (Optional)
  • Step 3: Run Your First Backup
• Usage and Examples
  • Quick One-Off Backup
  • Basic Backup
  • Portable Usage (Plug-and-Play)
  • Dry Run
  • One-Off Snapshot Backup
  • Excluding Files and Directories
  • Using Pre- and Post-Backup Hooks
• Automation
• How to Restore a Backup
  • 1. Locate Your Backup
  • 2. Extract Files from Archives
• Cross-Platform Backups and Case-Sensitivity
• Exclusion Rules
• Log Levels
• Retention Policy
  • Retention Policy Examples
• Configuration Details
• Troubleshooting
• Contributing
• Acknowledgments
• License

• Reliability: Pre-flight checks validate paths, permissions, and configuration before a single byte is moved.
• Transparency: No proprietary containers. Backups are stored exactly as they appear in the source or within standard archives.
• Performance: A high-performance, concurrent sync engine designed for modern hardware and high-latency network shares.
• Safety: Built-in protection against "ghost" mount points, permission lockouts, and case-sensitivity mismatches.

• Backup Modes:
  • Incremental (default): Maintains a single "current" backup directory that is efficiently updated. At a configurable interval (e.g., daily), the "current" backup is rolled over into a timestamped archive. This is ideal for frequent, low-overhead backups.
  • Snapshot: Each backup run creates a new, unique, timestamped directory. This is useful for creating distinct, point-in-time copies.
• Zero Vendor Lock-in: Your data is stored in standard directories and open archive formats (.zip, .tar.gz, .tar.zst). You can always access and restore your files using native operating system tools, even without pgl-backup installed.
• Flexible Retention Policy: Automatically cleans up outdated backups by keeping a configurable number of hourly, daily, weekly, monthly, and yearly archives. This gives you a fine-grained history without filling up your disk.
• Intelligent Archiving (Incremental Mode): The archive interval can be set to auto (the default) to automatically align with your retention policy. If you keep hourly backups, it will archive hourly. This prevents configuration mismatches and ensures your retention slots are filled efficiently.
• Automatic and Resilient Compression: To save disk space, you can enable automatic compression of backups into .tar.zst, .zip or .tar.gz archives. If compression fails (e.g., due to a corrupt file), the original backup is left untouched.
• Sleep Prevention: Automatically prevents the system from entering sleep or idle modes during active backup, restore, or prune operations. This ensures that long-running tasks are not interrupted by power management settings. Supports Windows, macOS, and Linux (via systemd-inhibit).
• HDD & Tape Drive Optimization: A dedicated -sync-sequential-write mode buffers files in memory before writing them to disk sequentially. This eliminates disk head thrashing on mechanical drives, significantly increasing write speeds during concurrent backups.
• Concurrency Safe: A robust file-locking mechanism prevents multiple backup instances from running against the same target directory simultaneously, protecting data integrity.
• Symbolic Link Support: Preserves symbolic links in the destination. On Windows, creating symlinks requires the user to have the appropriate privileges (Run as Administrator) or Developer Mode enabled.
• Plug and Play: The binary is static and self-contained. You can run it directly from an external drive without installation, making it perfect for portable backup workflows.
• Pre- and Post-Backup Hooks: Execute custom shell commands before the sync begins or after it completes, perfect for tasks like dumping a database or sending a notification.
• Adjustable Configuration: Configure backups using a simple pgl-backup.config.json JSON file, and override any setting with command-line flags for one-off tasks.
• High performance Sync Engine:
  • native: A high-performance, concurrent, cross-platform engine written in pure Go that offers the best performance with no external dependencies.
• Safety First:
  • Pre-flight Checks: Validates source and target paths, permissions, and configuration before starting any file operations to fail fast and provide clear errors.
  • Dry Run Mode: The -dry-run flag lets you see exactly what files would be copied, updated, or deleted without making any actual changes.
  • Permission Lockout Protection: When copying files and directories, pgl-backup automatically ensures the backup user has write permissions on the destination, even if the source is read-only. This prevents the backup process from locking itself out on subsequent runs.
  • Consistent User Execution: While pgl-backup includes features to prevent permission lockouts, it is still a best practice to run all backups for a specific target as the same user. This ensures consistent file ownership and avoids potential permission issues on Unix-like systems.
  • "Ghost Directory" Protection: On Unix-like systems, it helps prevent accidentally backing up to a mount point when the external drive is not actually mounted.
  • Cross-Platform Safety: Detects and halts on risky backup scenarios, such as backing up a case-sensitive Linux source from a case-insensitive Windows host, which can lead to silent data loss.
  • Path Nesting Protection: Prevents infinite recursion loops by ensuring source and target directories are not nested within each other.

pgl-backup is designed with data integrity as its highest priority, especially when the backup repository is being monitored by a cloud sync client (like Google Drive, Dropbox, or Synology Drive).

The backup process follows a robust "Stage, then Process" pipeline:

1. A new backup is created by syncing the source to a temporary current directory.
2. If an old backup is due for archival, it is moved to a temporary stage directory.
3. The staged backup is then compressed.
4. Finally, the fully processed backup is moved from stage into its permanent home in the archive directory using an os.Rename operation.

The use of os.Rename is critical. On most filesystems, a rename is an atomic operation. This means that from the perspective of the filesystem (and any application watching it), the backup directory appears instantly and completely in its final destination.

Sync clients will never see a partially copied or partially compressed backup. They will only ever see a complete, consistent backup directory appear in the archive, which they can then safely upload. This eliminates the risk of syncing an incomplete or corrupt backup state to the cloud, making the archive directories extremely safe to use as a source for a secondary cloud backup.

For cloud synchronization, you should point your sync client to the following directories inside your backup base path:

• PGL_Backup_Incremental_Archive
• PGL_Backup_Snapshot_Archive

Note: PGL stands for PixelGardenLabs, the namespace used for all internal and open-source projects in this ecosystem.

These are the default names for the directories containing the final, safe-to-sync backups. You can customize these names after running pgl-backup init by editing the paths section in your pgl-backup.config.json file.

Important: You should not sync the ..._Current or ..._Stage directories, as they contain in-progress or temporary data.

Ensure you have Go installed (version 1.26+ recommended). If you have a Go environment set up, you can install pgl-backup directly:

go install github.com/paulschiretz/pgl-backup@latest

You can download the latest pre-compiled binaries for your operating system from the Releases page.

As an open-source project, the pre-compiled binaries provided in the Releases section are currently unsigned. This means that operating systems like macOS and Windows may flag them as "unrecognized" or "untrusted" when you try to run them for the first time.

This is expected behavior for unsigned software. You can verify the integrity of the downloaded files using the provided checksums.txt file, or build the project from source if you prefer.

Note: At the current stage, binaries are unsigned to avoid the high costs of code signing certificates for a free open-source tool. If this is a significant issue for you, please open an issue on GitHub. I am willing to sign binaries in the future if there is sufficient demand to justify the cost.

When you first run pgl-backup on macOS, you may see a popup saying it "cannot be opened because the developer cannot be verified."

1. Locate the pgl-backup binary in Finder.
2. Right-click (or Control-click) the file and select Open.
3. Click Open in the dialog box that appears.
4. You only need to do this once.

Alternatively, you can remove the quarantine attribute via the terminal:

xattr -d com.apple.quarantine pgl-backup

Windows Defender SmartScreen may prevent the application from starting.

1. Click More info in the popup window.
2. Click the Run anyway button.

Every release includes a checksums.txt file containing the SHA256 hashes of the artifacts. You can verify that your download has not been tampered with or corrupted.

Linux / macOS:

# Download the checksums.txt and the archive to the same directory
shasum -a 256 -c checksums.txt

# Or manually check a specific file:
shasum -a 256 pgl-backup_v1.4.6_darwin_arm64.tar.gz
# Compare output with the content of checksums.txt

Windows (PowerShell):

Get-FileHash .\pgl-backup_v1.4.6_windows_amd64.zip -Algorithm SHA256
# Compare the hash with the content of checksums.txt

Let's set up a daily incremental backup for your ~/Documents folder to an external drive.

The easiest way to get started is to use the init command. This will generate a pgl-backup.config.json file in your target directory.

• New Backups: It creates the directory and a default configuration file. The -base and -source flags are required.
• Existing Backups: It reads your existing configuration, applies any new flags you provide (like changing the log level), and saves the updated config. It preserves your retention policies and other settings. Note that -source is required to perform pre-flight checks.
• Fresh Start: If you want to completely overwrite an existing configuration with defaults, use init -default instead.

Update Anytime: You can run the init command again at any time with new flags to update specific settings (e.g., exclusions or worker counts). It will merge your changes into the existing configuration file while preserving other settings.

# Example for Linux/macOS
pgl-backup init -base="/media/backup-drive/MyDocumentsBackup" -source="$HOME/Documents"

# Example for Windows
pgl-backup init -base="E:\Backups\MyDocumentsBackup" -source="C:\Users\YourUser\Documents"

HDD Optimization Tip: If your backup target is a mechanical hard drive (HDD) or a Tape-Drive, we highly recommend initializing with the -sync-sequential-write flag. This optimizes write patterns to prevent disk thrashing and significantly improves performance by buffering files in RAM and writing them one by one.

# Example for use with HDD drives
pgl-backup init -base="/path/to/your/backup-target" -source="/path/to/your/source-data" -sync-sequential-write

You can also combine the init command with other flags to customize the configuration immediately:

# Example for Linux/macOS
pgl-backup init -base="/media/backup-drive/MyDocumentsBackup" -source="$HOME/Documents" -log-level=debug -user-exclude-files="*.mp4"

# Example for Windows
pgl-backup init -base="E:\Backups\MyDocumentsBackup" -source="C:\Users\YourUser\Documents" -log-level=debug -user-exclude-files="*.mp4"

Note on Boolean Flags: Boolean flags(enable/disable) (like -metrics, -sync-sequential-write or -sync-safe-copy) can be enabled by simply providing the flag (e.g., -metrics). To explicitly disable them, use the = syntax (e.g., -metrics=false).

This command will:

1. Perform pre-flight checks to ensure the paths are valid.
2. Create the target directory /media/backup-drive/MyDocumentsBackup.
3. Generate a pgl-backup.config.json file inside it.

Open the newly created pgl-backup.config.json file. It will look something like this, filled with sensible defaults. You can adjust the retention policy, archive policy, or add exclusions.

{
  "version": "1.4.6",
  "logLevel": "info",
  "paths": {
    "incremental": {
      "current": "PGL_Backup_Incremental_Current",
      "archive": "PGL_Backup_Incremental_Archive",
      "archiveEntryPrefix": "PGL_Backup_",
      "stage": ".~PGL_Backup_Incremental_Stage",
      "stageEntryPrefix": "PGL_Backup_",
      "content": "PGL_Backup_Content"
    },
    "snapshot": {
      "current": "PGL_Backup_Snapshot_Current",
      "archive": "PGL_Backup_Snapshot_Archive",
      "archiveEntryPrefix": "PGL_Backup_",
      "stage": ".~PGL_Backup_Snapshot_Stage",
      "stageEntryPrefix": "PGL_Backup_",
      "content": "PGL_Backup_Content"
    }
  },
  "engine": {
    "metrics": true,
    "failFast": false,
    "performance": {
      "syncWorkers": 4,
      "mirrorWorkers": 4,
      "deleteWorkers": 4,
      "compressWorkers": 4,
      "bufferSizeKB": 1024,
      "readAheadLimitKB": 262144
    }
  },
  "sync": {
    "enabled": true,
    "preserveSourceDirName": true,
    "engine": "native",
    "safeCopy": true,
    "sequentialWrite": false,
    "retryCount": 3,
    "retryWaitSeconds": 5,
    "modTimeWindowSeconds": 1,
    "defaultExcludeFiles": [
      "*.tmp",
      "*.temp",
      "*.swp",
      "*.lnk",
      "~*",
      "desktop.ini",
      ".DS_Store",
      "Thumbs.db",
      "Icon\r"
    ],
    "defaultExcludeDirs": [
      "@tmp",
      "@eadir",
      ".SynologyWorkingDirectory",
      "#recycle",
      "$Recycle.Bin"
    ],
    "userExcludeFiles": [],
    "userExcludeDirs": []
  },
  "archive": {
    "enabled": true,
    "intervalMode": "auto",
    "intervalSeconds": 86400
  },
  "retention": {
    "incremental": {
      "enabled": true,
      "hours": 0,
      "days": 0,
      "weeks": 4,
      "months": 0,
      "years": 0
    },
    "snapshot": {
      "enabled": false,
      "hours": -1,
      "days": -1,
      "weeks": -1,
      "months": -1,
      "years": -1
    }
  },
  "compression": {
    "enabled": true,
    "format": "tar.zst",
    "level": "default"
  },
  "hooks": {
    "enabled": false,
    "preBackup": [],
    "postBackup": [],
    "preRestore": [],
    "postRestore": []
  }
}

Now, simply point pgl-backup at the target directory and provide the source. It will automatically load the configuration file from the base directory and run the backup. The -source flag is mandatory.

# Run the backup
pgl-backup backup -base="/media/backup-drive/MyDocumentsBackup" -source="$HOME/Documents"

Command Structure:

• -base(Required): The root directory of your backup repository (where pgl-backup.config.json is located).
• -source(Required): The source directory where the files you want to backup are stored.

The first run will copy all files into a PGL_Backup_Content subdirectory inside the main PGL_Backup_Incremental_Current directory. Subsequent runs will efficiently update this directory.

When it's time to create a historical archive (e.g., after a week), the next backup run will:

1. Move the old PGL_Backup_Incremental_Current to a temporary staging area.
2. Create a new, up-to-date PGL_Backup_Incremental_Current by syncing from your source.
3. Clean up old backups from the archive according to your retention policy.
4. Compress the staged backup (if enabled) and move it into the PGL_Backup_Incremental_Archive directory as a permanent, timestamped record.

This robust Stage -> Prune -> Archive flow ensures that old backups are removed before new ones are added, preventing "disk full" errors and keeping the main archive clean.

Your backup target will be organized like this:

/path/to/your/backup-target/
├── pgl-backup.config.json (The configuration for this backup set)
├── PGL_Backup_Incremental_Current/ (The current incremental backup)
├── PGL_Backup_Incremental_Archive/ (Timestamped historical incremental backups)
└── PGL_Backup_Snapshot_Archive/ (Timestamped backups from snapshot mode)

pgl-backup uses a subcommand structure: pgl-backup <command> [flags]. Running pgl-backup without any arguments will display the help message.

Although recommended, you don't need to generate a configuration file with init to run a backup. You can simply provide the necessary flags directly. pgl-backup will use sensible defaults for everything else.

pgl-backup backup -base="/path/to/your/backup-target" -source="/path/to/your/source-data"

HDD Optimization Tip: If your backup target is a mechanical hard drive (HDD) or a Tape-Drive, add the -sync-sequential-write flag. This optimizes write patterns to prevent disk thrashing and significantly improves performance by buffering files in RAM and writing them one by one.

pgl-backup backup -base="..." -source="..." -sync-sequential-write

Once configured, this is the only command you need for your backup script, cron job or scheduled task.

pgl-backup backup -base="/path/to/your/backup-target" -source="/path/to/your/source-data"

You can store the pgl-backup binary directly on your external backup drive alongside your data. This allows you to plug the drive into any computer and run backups immediately without installing any software.

For example, you can place the binary and a simple script on your external drive:

Directory Structure:

X:\ (External Drive)
├── pgl-backup.exe (Windows binary)
├── pgl-backup (Linux/macOS binary)
├── run_backup.bat
├── run_backup.sh
└── MyBackups\

Windows (run_backup.bat):

:: Set working directory to where the .bat file is
cd /d "%~dp0"
:: Run pgl-backup using a relative path for the base
pgl-backup.exe backup -base=".\MyBackups" -source="C:\Users\YourUser\Documents"
pause

Linux / macOS (run_backup.sh):

#!/bin/sh
cd "$(dirname "$0")"
./pgl-backup backup -base="./MyBackups" -source="$HOME/Documents"

See what changes would be made without touching any files.

pgl-backup backup -base="/path/to/your/backup-target" -source="/path/to/your/source-data" -dry-run

Here's how to perform a single snapshot backup.

pgl-backup backup -base="/path/to/your/backup-target" -source="/path/to/your/source-data" -mode=snapshot

You can define a list of files and directories to exclude from your backups. This is useful for ignoring temporary files, caches, or large media files. Patterns support standard file globbing.

You can set or update exclusions in two ways:

1. Using the init command: Use the -user-exclude-* flags with the init command to generate/update a configuration with your exclusions.

pgl-backup init -base="..." -source="..." -user-exclude-files="*.tmp,*.log" -user-exclude-dirs="node_modules,.cache"

2. By Editing the Configuration File: Alternatively, you can open pgl-backup.config.json at any time and add your patterns to the sync section:

"sync": {
  ...
  "userExcludeFiles": [
    "*.tmp",
    "*.log"
  ],
  "userExcludeDirs": [
    "node_modules",
    ".cache"
  ]
}

Your next backup run will automatically apply these exclusions.

Note on Matching: All exclusion patterns are case-insensitive on all operating systems. A pattern like *.jpeg will match photo.jpeg, photo.JPEG, and photo.JpEg. This ensures your configuration is portable and behaves predictably across Windows, macOS, and Linux.

You can configure scripts to run before or after operations. Define these commands during initialization or in your config file. Commands with spaces must be wrapped in single or double quotes. Note: Hooks are disabled by default for security. You must explicitly enable them using the -hooks flag or by setting "enabled": true in the configuration.

# Configure hooks during init
pgl-backup init -base="..." -source="..." -hooks-pre-backup="'/usr/local/bin/dump_database.sh', 'echo Backup starting...'"

# Enable hooks during backup (disabled by default for security)
pgl-backup backup -base="..." -source="..." -hooks

Security Note: Hooks execute arbitrary shell commands. Ensure that any commands in your configuration are from a trusted source and have the correct permissions to prevent unintended side effects.

Manually apply retention policies to clean up outdated backups without running a full backup. This is useful for freeing up disk space immediately. Supports -dry-run to preview deletions.

pgl-backup prune -base="/path/to/your/backup-target"

Command Structure:

• -base(Required): The root directory of your backup repository (where pgl-backup.config.json is located).
• -mode: Specify the backup mode (incremental or snapshot). If omitted, pgl-backup automatically prunes incremental and snapshot backups.

View a list of all backups stored in the repository, including their timestamps, modes, and UUIDs. This is useful for finding the specific name of a backup you want to restore.

pgl-backup list -base="/path/to/your/backup-target"

Command Structure:

• -base(Required): The root directory of your backup repository (where pgl-backup.config.json is located).
• -mode: Specify the backup mode (incremental or snapshot). If omitted, pgl-backup automatically lists incremental and snapshot backups.
• -sort: Sort order for the list (desc or asc). Defaults to desc (newest first).

You can easily automate your backups using your operating system's built-in scheduler.

1. Open Task Scheduler and click Create Basic Task.
2. Name it "Daily Backup" and click Next.
3. Choose your trigger (e.g., Daily) and set the time.
4. Select Start a program as the action.
5. Program/script: Browse to your pgl-backup.exe.
6. Add arguments: Enter your backup command flags.

   backup -base="D:\Backups" -source="C:\Users\YourName\Documents"
   

7. Finish the wizard.

Edit your crontab file (crontab -e) and add a line to run the backup daily (e.g., at 2:00 AM):

0 2 * * * /usr/local/bin/pgl-backup backup -base="/mnt/backup-drive" -source="/home/yourname/Documents" >> /var/log/pgl-backup.log 2>&1

If your backup target is an external drive and it is not connected when the scheduled task runs, pgl-backup will detect this during its pre-flight checks and exit safely with an error. No data will be lost, and the backup will simply be skipped until the next scheduled run when the drive is available.

pgl-backup provides two ways to restore your data: using the built-in restore command for convenience, or by manually accessing your files with standard OS tools. The manual method guarantees you can always recover your data, even without pgl-backup installed.

The restore command is the easiest and safest way to restore a backup. It automatically handles both compressed archives and uncompressed directories.

The restore command copies data from a specific backup to a target directory.

pgl-backup restore -base="/path/to/backup-repo" -target="/path/to/restore-location"

Command Structure:

• -base(Required): The root directory of your backup repository (where pgl-backup.config.json is located).
• -target(Required): The directory where you want to restore the files.
• -uuid: The UUID of the backup you want to restore. If omitted (the default), pgl-backup launches an easy-to-use interactive menu where you can browse and select a backup from a list.
• -mode: Filter the interactive list by backup mode (incremental or snapshot), or restrict the search when -uuid is provided. If omitted (any), pgl-backup lists all backups or searches both locations.
• -overwrite: Control overwrite behavior (always, never, if-newer, update). If omitted, defaults to never, so existing files are not overwritten.
• -fail-fast: Stop immediately on the first error. If omitted, defaults to false.

Examples:

1. Interactive Restore (Default & Easiest): Simply omit the -uuid flag to see a list of all available backups and select one by number.

pgl-backup restore -base="/media/backup-drive/MyDocumentsBackup" \
                   -target="$HOME/RestoredDocuments"

The list displays the backup type:

• [INC]: Incremental backup.
• [SNP]: Snapshot backup

2. Restore a specific backup by UUID: This finds the backup with the specified UUID, automatically decompresses it if needed, and restores its contents. You can find the UUID using the list command.

pgl-backup restore -base="/media/backup-drive/MyDocumentsBackup" \
                   -target="$HOME/RestoredFromSnapshot" \
                   -uuid="123e4567-e89b-12d3-a456-426614174000" \

3. Restore the latest backup: Automatically selects and restores the most recent backup.

pgl-backup restore -base="/media/backup-drive/MyDocumentsBackup" \
                   -target="$HOME/RestoredFromSnapshot" \
                   -uuid="latest" \

4. Restore a specific archived snapshot: This finds the backup with the specified UUID inside the snapshot archive directory, automatically decompresses it if needed, and restores its contents.

pgl-backup restore -base="/media/backup-drive/MyDocumentsBackup" \
                   -target="$HOME/RestoredFromSnapshot" \
                   -uuid="123e4567-e89b-12d3-a456-426614174000" \
                   -mode="snapshot"

Overwrite Behavior: By default, the restore command will never overwrite existing files in the target directory (-overwrite=never). You can change this behavior:

• -overwrite=if-newer: Only overwrite if the file in the backup is newer.
• -overwrite=always: Always overwrite existing files.

A core design philosophy of pgl-backup is zero vendor lock-in. Your data is always accessible.

Navigate to your backup target directory. You will see the following structure:

• PGL_Backup_Incremental_Current/: Contains the most recent version of your files within the PGL_Backup_Content subdirectory. You can browse this directory directly and copy files out using your file explorer.
• PGL_Backup_Incremental_Archive/: Contains compressed historical incremental backups (e.g., PGL_Backup_2023-10-27-14-00-00...tar.zst), if compression is disabled all your files are within the PGL_Backup_Content subdirectory.
• PGL_Backup_Snapshot_Archive/: Contains point-in-time snapshots if you use snapshot mode. If compression is enabled (e.g., PGL_Backup_2023-10-27-14-00-00...tar.zst), if compression is disabled all your files are within the PGL_Backup_Content subdirectory.

If you need to restore files from a compressed archive in PGL_Backup_Incremental_Archive or PGL_Backup_Snapshot_Archive, use the standard tools for your operating system.

Tip: For a consistent graphical experience across platforms, or if you prefer not to use the command line, tools like 7-Zip (Windows) or PeaZip (Windows, Linux, macOS) can handle .zip, .tar.gz, and .tar.zst archives effortlessly.

1. Right-click the .zip file.
2. Select Extract All....
3. Choose a destination and click Extract.

You can double-click the file to extract it, or use the terminal:

# Extract the entire archive
tar -xvf PGL_Backup_2023-10-27.tar.gz

# Extract a specific file
tar -xvf PGL_Backup_2023-10-27.tar.gz "path/to/file.txt"

The .tar.zst format uses Zstandard compression for high speed and ratio. You may need to install zstd (e.g., brew install zstd or apt install zstd).

# Extract the entire archive
tar --zstd -xvf PGL_Backup_2023-10-27.tar.zst

# Extract a specific file
tar --zstd -xvf PGL_Backup_2023-10-27.tar.zst "path/to/file.txt"

pgl-backup includes a critical safety check to prevent data loss from filesystem case-sensitivity mismatches.

• The Problem: Backing up a case-sensitive source (like a Linux filesystem with both File.txt and file.txt) from a case-insensitive host (like Windows) is dangerous. The host OS may only "see" one of the files, leading to silent data loss as the other is never backed up.
• The Solution: pgl-backup automatically tests the case-sensitivity of both the source and the host environment. If it detects this risky combination, it will stop with a clear error message before the backup begins.
• Recommendation: For maximum data integrity when backing up a case-sensitive source (like a Linux server or a WSL environment), you should always run pgl-backup on a case-sensitive operating system (like Linux, or from within WSL on Windows).

pgl-backup provides a flexible exclusion system to keep your backups clean and efficient.

All exclusion patterns are case-insensitive on all operating systems. A pattern like *.jpeg will match photo.jpeg, photo.JPEG, and photo.JpEg. This ensures your configuration is portable and behaves predictably across Windows, macOS, and Linux.

To provide a better out-of-the-box experience and protect its own metadata, pgl-backup uses two layers of exclusions.

A small, non-configurable list of files are always excluded to prevent the backup tool from backing up its own operational files. These are:

• pgl-backup.config.json
• .pgl-backup.meta.json
• .~pgl-backup.lock

A list of common temporary and system files are excluded by default. When you generate a new configuration file with -init, these defaults are included, and you can customize them in your pgl-backup.config.json file. The JSON keys for these are defaultExcludeFiles and defaultExcludeDirs.

• Default Excluded Files: *.tmp, *.temp, *.swp, *.lnk, ~*, desktop.ini, .DS_Store, Thumbs.db, Icon\r.
• Default Excluded Directories: @tmp, @eadir, .SynologyWorkingDirectory, #recycle, $Recycle.Bin.

You can define your own list of files and directories to exclude using the userExcludeFiles and userExcludeDirs keys in the configuration file, or via the command-line flags -user-exclude-files and -user-exclude-dirs of the init command. These are combined with the system and default exclusions.

pgl-backup uses a structured logging system with several levels of verbosity, allowing you to control how much detail you see. You can set the level using the -log-level flag or the logLevel key in your configuration file.

• error: Only shows critical errors that cause the backup process to halt. Use this if you only want to be alerted to complete failures.
• warn: Shows errors and warnings. Warnings are non-critical issues that the application has recovered from but that you should be aware of (e.g., a single file failed to copy, a configuration mismatch).
• info (Default): Provides a high-level summary of the backup process. It shows the start, periodic progress, and end of major phases like synchronization, compression, and retention cleanup. This is the recommended level for daily use and cron jobs, as it provides a clean, readable overview.
• notice: Shows everything from info plus detailed, per-item operational messages. Use this level to see a log of every file being copied, deleted, archived, or compressed. It's useful for verifying that specific files are being handled correctly without the full verbosity of debug.
• debug: The most verbose level. Includes everything from notice plus detailed developer-oriented information for tracing execution flow and diagnosing complex issues.

The retention policy is designed to give you a detailed short-term history and a space-efficient long-term history. pgl-backup supports separate retention policies the Incremental mode archive and Snapshot mode archive.

• Incremental Retention: Designed for your routine backup history. Since incremental backups run frequently (e.g., daily), this policy thins them out over time to save space. It is enabled by default.
• Snapshot Retention: Designed for manual milestones (e.g., "Pre-Upgrade"). By default, this policy is disabled, meaning snapshots are kept forever. If you enable it, you must explicitly set retention periods to 0 or greater (defaults are -1 for safety).

Both policies work using a "promotion" system. When cleaning up old backups, pgl-backup scans all your archived backups from newest to oldest and decides which ones to keep.

Here's how it works for each backup, in order of priority:

1. Hourly: Is there an open "hourly" slot? If yes, keep this backup and move to the next one.
2. Daily: If not kept as hourly, is there an open "daily" slot for this backup's calendar day? If yes, keep it and move on.
3. Weekly: If not kept as daily, is there an open "weekly" slot for this backup's calendar week? If yes, keep it and move on.
4. Monthly: If not kept as weekly, is there an open "monthly" slot? If yes, keep it and move on.
5. Yearly: If not kept as monthly, is there an open "yearly" slot? If yes, keep it.

If a backup doesn't fill any available slot, it is deleted. This ensures that a backup is only "promoted" to a longer-term slot (like weekly) if it's no longer needed to fill a shorter-term slot (like daily).

Here are a few example policies you can use in your pgl-backup.config.json file. The best policy depends on how much data you are backing up and how much disk space you have available. For many use cases, a simple policy of keeping 4 weekly backups is more than enough.

Goal: A minimal policy for users who need a basic safety net without using much disk space. It provides a few recent recovery points. Total Backups Stored: ~4 (2 daily + 1 weekly + 1 monthly) Auto Archive Interval Sets To: 24 Hours

"retention": {
  "incremental": {
    "enabled": true,
    "hours": 0,
    "days": 2,
    "weeks": 1,
    "months": 1,
    "years": 0
  }
}

Goal: A simple, "set-it-and-forget-it" policy that provides a month of history without using excessive disk space. This is the default policy when you first initialize pgl-backup. Total Backups Stored: ~4 (4 weekly) Auto Archive Interval Sets To: 1 Week

"retention": {
  "incremental": {
    "enabled": true,
    "hours": 0,
    "days": 0,
    "weeks": 4,
    "months": 0,
    "years": 0
  }
}

Goal: You mostly care about the current state (which is always available in PGL_Backup_Incremental_Current) but want one previous snapshot just in case. Total Backups Stored: ~1 (1 monthly) Auto Archive Interval Sets To: ~30 Days

"retention": {
  "incremental": {
    "enabled": true,
    "hours": 0,
    "days": 0,
    "weeks": 0,
    "months": 1,
    "years": 0
  }
}

Goal: For data that changes slowly. You only create an archive once a month. Total Backups Stored: ~12 (12 monthly) Auto Archive Interval Sets To: 30 Days

"retention": {
  "incremental": {
    "enabled": true,
    "hours": 0,
    "days": 0,
    "weeks": 0,
    "months": 12,
    "years": 0
  }
}

Goal: Protect active work where recent recovery is critical. The priority is being able to undo a mistake made recently. Total Backups Stored: ~6 (3 hourly + 3 daily) Auto Archive Interval Sets To: 1 Hour

"retention": {
  "incremental": {
    "enabled": true,
    "hours": 3,
    "days": 3,
    "weeks": 0,
    "months": 0,
    "years": 0
  }
}

Goal: A robust policy for important data. It balances a reasonable safety net (3 days) with a solid historical archive (3 months). Total Backups Stored: ~10 (3 daily + 4 weekly + 3 monthly) Auto Archive Interval Sets To: 24 Hours

"retention": {
  "incremental": {
    "enabled": true,
    "hours": 0,
    "days": 3,
    "weeks": 4,
    "months": 3,
    "years": 0
  }
}

Goal: For data where you might need to go back many years, but fine-grained daily history isn't needed. Total Backups Stored: ~11 (4 weekly + 7 yearly) Auto Archive Interval Sets To: 1 Week

"retention": {
  "incremental": {
    "enabled": true,
    "hours": 0,
    "days": 0,
    "weeks": 4,
    "months": 0,
    "years": 7
  }
}

Goal: Backing up terabytes of movies/photos where data rarely changes and storage costs are high. You just want a few recent versions in case of accidental deletion. Total Backups Stored: ~3 (1 weekly + 1 monthly + 1 yearly) Auto Archive Interval Sets To: 1 Week

"retention": {
  "incremental": {
    "enabled": true,
    "hours": 0,
    "days": 0,
    "weeks": 1,
    "months": 1,
    "years": 1
  }
}

• backup: Run the backup operation. Use -help to see all the options.
• restore: Run the restore operation. Use -help to see all the options.
• list: List all backups in the base directory. Use -help to see all the options.
• prune: Apply retention policies to clean up outdated backups. Use -help to see all the options.
• init: Initialize or update a configuration. Use -default to overwrite an existing configuration with defaults.
• version: Print the application version.

All command-line flags can also be set in the pgl-backup.config.json file. Note that structural options (like directory paths) are only available in the configuration file to ensure consistency.

Note on Boolean Flags: Flags with type bool(true/false) can be enabled by simply passing the flag name (e.g., -dry-run is equivalent to -dry-run=true). To disable a boolean flag that defaults to true, you must explicitly set it to false using the equals sign (e.g., -sync-safe-copy=false).

This occurs because the binaries are currently unsigned. Please refer to the Security and Binary Verification section for instructions on how to verify and run the application on macOS and Windows.

• Cause: The user running pgl-backup does not have the necessary read permissions for the source directory or write permissions for the target directory.
• Solution:
  • Ensure you are running the command as a user with appropriate permissions.
  • On Unix-like systems, use ls -l to check ownership and permissions. You may need to use sudo or adjust file permissions with chmod and chown.
  • For network shares (SMB/NFS), verify that the share is mounted correctly and that the user has write access.

• Cause: Another pgl-backup process is currently running against the same target directory, or a previous run crashed without cleanly releasing the lock. The application has stale-lock detection, but it may not cover all edge cases (e.g., system clock changes).
• Solution:
  1. First, verify that no other pgl-backup process is running. Use ps aux | grep pgl-backup (Linux/macOS) or check Task Manager (Windows).
  2. If you are certain no other process is active, the lock file is stale. You can safely delete the .~pgl-backup.lock file from the root of your target backup directory and re-run the command.

• Cause: The command or script specified in the hook may not be executable, not in the system's PATH, or has incorrect quoting.
• Solution:
  • Ensure that any script you are calling is marked as executable (e.g., chmod +x /path/to/your/script.sh).
  • Use absolute paths for your scripts (e.g., /usr/local/bin/my_script.sh) to avoid PATH issues, especially in automated environments like cron.
  • If your command contains spaces or special characters, ensure it is correctly quoted within the JSON configuration or on the command line (e.g., "'echo Backup starting...'", "/path/to/script with spaces.sh").

• Cause: The default number of concurrent workers may not be optimal for your specific hardware (e.g., slow spinning disks vs. fast SSDs, network latency).
• Solution:
  • Adjust the sync-workers and buffer-size-kb settings in your pgl-backup.config.json file.
  • For systems with slow I/O (like a single spinning disk or a high-latency network share), decreasing the number of sync-workers (e.g., to 1 or 2) is recommended to minimize seek latency, as higher concurrency can cause significant thrashing.
  • For systems with very fast I/O (like a local NVMe SSD), increasing sync-workers might yield better results.
  • HDD Optimization: If writing to a mechanical hard drive, enable "sequentialWrite": true (or -sync-sequential-write). This buffers files in memory (up to readAheadLimitKB) and writes them sequentially, preventing the disk head from thrashing between multiple concurrent writes. This allows you to keep the sync-workers count high (e.g., 4) to maximize read throughput from the source while keeping writes linear.
• Cause: The default "Safe Copy" mechanism ensures atomicity but increases metadata operations (create temp file + rename). This can be slow on high-latency network shares or with many small files.
• Solution:
  • Try disabling Safe Copy by setting "safeCopy": false in your config or using -sync-safe-copy=false. This switches to direct writes, improving performance at the cost of atomicity (a power loss during copy might leave a partial file, which will be fixed on the next run).

• Cause: The backup source contains symbolic links, but the user running pgl-backup on Windows does not have permission to create them in the destination.
• Solution:
  • Run the command prompt or PowerShell as Administrator.
  • Enable Developer Mode in Windows settings, which allows non-admins to create symlinks.

• Cause: The pgl-backup.config.json file contains invalid JSON syntax (e.g., a missing comma, a trailing comma after the last item in a list, or unescaped backslashes in paths).
• Solution:
  • Validate your JSON content using an online JSON validator.
  • Ensure backslashes in Windows paths are escaped (e.g., "C:\\Users\\Name" instead of "C:\Users\Name") or use forward slashes ("C:/Users/Name"), which work on all platforms.

• Cause: The file is currently open and locked by another application (e.g., an open Word document, Outlook .pst file, or running database).
• Note: pgl-backup does not use the Volume Shadow Copy Service (VSS). VSS is a complex, Windows-only technology that requires Administrator privileges and is typically only needed for specific edge cases. pgl-backup prioritizes simplicity, speed, and cross-platform consistency.
• Solution:
  • Close the application using the file before running the backup.
  • Use the hooks-pre-backup feature to stop the relevant service/application before the backup and hooks-post-backup to restart it.

• Cause: This often happens when backing up to a file system with lower timestamp precision (like FAT32/exFAT) or a network share (SMB/CIFS) where the server time drifts from the client time.
• Solution:
  • Increase the modTimeWindowSeconds setting in your configuration (e.g., to 2 or 5 seconds). This tells pgl-backup to treat timestamps as identical if they are within that window.

Please see our Contributing Guidelines for details on how to contribute to the project.

• Special thanks to Klaus Post for his excellent compression libraries (github.com/klauspost/compress), which enable the high-performance compression features in pgl-backup.
• The Go Authors for the Go programming language and standard library.

This project is licensed under the MIT License. See the LICENSE file for full license text.

This software includes components from third-party projects and the Go standard library. See the NOTICE file for third-party attributions and full license texts.