Skip to content

Log audit phase 1#1539

Merged
brianmcgillion merged 5 commits intotiiuae:mainfrom
juliuskoskela:log-audit-phase-1
Feb 3, 2026
Merged

Log audit phase 1#1539
brianmcgillion merged 5 commits intotiiuae:mainfrom
juliuskoskela:log-audit-phase-1

Conversation

@juliuskoskela
Copy link
Copy Markdown
Contributor

@juliuskoskela juliuskoskela commented Nov 6, 2025

Description of Changes

Implements Phase 1 – Forward Secure Sealing (FSS) for the Ghaf logging subsystem, establishing cryptographic tamper-evidence and audit-log integrity for all systemd journal entries.

This phase provides the foundation for verifiable audit trails by enabling and managing systemd’s built-in Forward Secure Sealing mechanism.

Key Additions

  • modules/common/logging/fss.nix

    • New ghaf.logging.fss option set (enable, keyPath, sealInterval, verifyOnBoot, verifySchedule).

    • Adds hardened systemd units:

      • journal-fss-setup.service – Generates FSS keys on first boot, extracts and stores the verification key under /persist/common/journal-fss.
      • journal-fss-verify.service + timer – Performs periodic and boot-time integrity verification, reporting via journald (AUDIT_LOG_VERIFY_COMPLETED / AUDIT_LOG_INTEGRITY_FAIL).

    • Integrates tmpfiles + auditd rules to monitor FSS key paths and sealed journals.

  • logging/client.nix / logging/server.nix

    • Enables Seal=yes in journald configuration when FSS is active.

  • logging/default.nix

    • Includes the new FSS module in the logging stack.

  • systemd/base.nix

    • Enables withGcrypt to ensure systemd is built with libgcrypt support (required for FSS HMAC chains).

Highlights

  • Fully NixOS-native, automatic initialization on first boot.
  • Tamper-evident audit logs sealed with rotating HMAC keys.
  • Offline-verifiable public verification key export.
  • Strictly sandboxed systemd services (no network, no new privileges).
  • Auditd monitoring of FSS key and journal directories.

Type of Change

  • New Feature
  • Bug Fix
  • Improvement / Refactor

Related Issues / Tickets

Implements Phase 1 of the Audit Log Integrity and Sealing roadmap.
Forms the baseline for Phase 2, which will introduce:

  • TPM-backed sealing keys generated and protected by the host TPM.
  • Certificate Authority (CA) attestation of verification keys for independent validation.
  • Optional remote verification API for fleet-wide integrity checks.

Checklist

  • Clear summary in PR description
  • Meaningful commit messages
  • Commits logically organized
  • Contribution guidelines followed
  • Documentation update planned (ghaf-docs/logging)
  • make-checks passes locally
  • All CI checks pass
  • Reviewers added, PR ready for review

Testing Instructions

Applicable Targets

  • Lenovo X1 x86_64
  • Dell Latitude x86_64
  • Orin AGX aarch64 (pending validation)

Installation Method

  • Can be updated with nixos-rebuild switch
  • Requires full re-installation

Test Steps to Verify

  1. Enable FSS in configuration:

    ghaf.logging.fss.enable = true;

  2. Rebuild system:

    sudo nixos-rebuild switch

  3. Reboot and confirm setup service:

    systemctl status journal-fss-setup.service

  4. Verify sealing active:

    journalctl --verify

    Expected: No errors found.

  5. Check periodic verification timer:

    systemctl list-timers | grep journal-fss-verify

  6. Inspect audit events:

    ausearch -k journal_fss_keys  
ausearch -k journal_sealed_logs

Next Phase (Planned): Integrate TPM 2.0 for hardware-rooted key sealing and CA-signed attestation of verification keys to enable cryptographically provable audit integrity across trusted nodes.

@brianmcgillion
Copy link
Copy Markdown
Collaborator

brianmcgillion commented Nov 6, 2025

@juliuskoskela very nice.

Copilot AI reviewed Nov 6, 2025
View reviewed changes
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This pull request implements Forward Secure Sealing (FSS) for systemd journal logs to provide cryptographic tamper-evidence for audit logs in the Ghaf framework. FSS uses HMAC-based sealing chains to detect any tampering with journal files.

Key changes:

  • Adds a new fss.nix module that configures FSS key generation and periodic journal verification
  • Enables libgcrypt in systemd (required for FSS support)
  • Configures Seal=yes in journald when FSS is enabled for both client and server logging

Reviewed Changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
modules/common/systemd/base.nix Enables libgcrypt support in systemd package (required for FSS)
modules/common/logging/fss.nix New module implementing FSS key setup, verification services, and audit monitoring
modules/common/logging/default.nix Imports the new FSS module
modules/common/logging/client.nix Enables journal sealing when FSS is enabled
modules/common/logging/server.nix Enables journal sealing when FSS is enabled

brianmcgillion
brianmcgillion requested changes Nov 6, 2025
View reviewed changes
Copy link
Copy Markdown
Collaborator

@brianmcgillion brianmcgillion left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

overall LGTM

@milva-unikie
Copy link
Copy Markdown

milva-unikie commented Nov 6, 2025

There are some failures in the pre-merge tests for laptops

  • No logs are being sent to Grafana
  • Following services are failing in all VMs and on the ghaf-host: [journal-fss-setup.service, journal-fss-verify.service, systemd-journal-flush.service]

@brianmcgillion
Copy link
Copy Markdown
Collaborator

brianmcgillion commented Dec 16, 2025

looks like you merged and not rebased.

everton-dematos
everton-dematos reviewed Dec 16, 2025
View reviewed changes
Copy link
Copy Markdown
Contributor

@everton-dematos everton-dematos left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you squash the commits? Maybe into 1 or 2... you choose how many 😃

@juliuskoskela
logging: implement Forward Secure Sealing for audit log integrity
dd5c694 
Implement Forward Secure Sealing (FSS) for systemd journal logs to provide
cryptographic tamper-evidence for audit logs. FSS uses HMAC-SHA256 chains
to seal journal entries at regular intervals, making any tampering detectable.

Key features:
- Per-component FSS key pairs (host and each VM have independent keys)
- Automatic key generation on first boot via journal-fss-setup service
- Periodic integrity verification via journal-fss-verify timer (hourly)
- Verification on boot to detect tampering during offline periods
- Audit rules to monitor FSS key and journal access
- Integration with existing logging infrastructure

Changes:
- Add FSS module (modules/common/logging/fss.nix) with setup/verify services
- Propagate logging.enable to all VMs for consistent FSS support
- Enable audit subsystem when FSS is enabled (provides auditctl)
- Add fss-test diagnostic tool to debug images
- Add FSS test suite for CI validation
- Add ADR and user documentation for FSS

The FSS implementation ensures that any modification to sealed journal
entries breaks the HMAC chain and is detected during verification,
providing strong tamper-evidence for security-critical audit logs.

Signed-off-by: Julius Koskela <julius.koskela@unikie.com>
@milva-unikie

This comment was marked as outdated.

Sign in to view
@juliuskoskela
logging: improve FSS verification and propagate listener config
86f7c16 
Improve journal FSS verification to handle user journal failures
separately from system journal failures. User journals may fail
verification due to entries written before FSS initialization,
which is expected and does not indicate tampering.

Also propagate the listener configuration to VM modules and add
missing mock option for tests.

Signed-off-by: Julius Koskela <julius.koskela@unikie.com>
@juliuskoskela
logging: fix FSS journal rotation after key setup
722f80e 
Per systemd documentation, journalctl --rotate is required after
--setup-keys to create a new journal file that uses the FSS keys.
Without rotation, entries in the existing journal remain unsealed.

Changes:
- Add journalctl --rotate after FSS key generation
- Update test script to use verification key for sealed journals
- Simplify service dependencies (no socket ordering needed)

Signed-off-by: Julius Koskela <julius.koskela@unikie.com>
@juliuskoskela
logging: fix FSS test key discovery for dynamic hostnames
7651330 
The FSS test script used $(hostname) to locate verification keys and
sentinel files. On net-vm, the dynamic hostname service overwrites the
hostname for AD integration, causing a mismatch with the Nix build-time
path used by journal-fss-setup.

Write an fss-config pointer file from the setup script into
/var/log/journal/<machine-id>/fss-config containing the Nix-configured
key directory path. The test script reads this pointer first and falls
back to hostname-based discovery for backward compatibility.

Also separate user journal failures from system journal failures in the
test script, treating user journal verification failures as warnings
since they contain pre-FSS entries.

Signed-off-by: Julius Koskela <julius.koskela@unikie.com>
@juliuskoskela
logging: fix FSS setup idempotency, journal rotation, and test timer …
9b1ee12 
…detection

Remove ConditionPathExists guard from setup service so it can re-run on
boot to write fss-config on existing deployments where the file was never
created. Add one-time journal rotation after FSS key setup to move
pre-FSS entries to archive, fixing EBADMSG ("Bad message") during
verification. Add systemctl cat fallback to test 6 for timer detection
in microVMs where list-unit-files may not show the timer.

Signed-off-by: Julius Koskela <julius.koskela@unikie.com>
@milva-unikie
Copy link
Copy Markdown

milva-unikie commented Feb 3, 2026

sudo fss-test passes in every VM on Darter Pro and on Lenovo X1.

journal-fss-verify.service failures have been fixed.

@milva-unikie milva-unikie mentioned this pull request Feb 3, 2026
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code review Copilot Copilot left review comments

@brianmcgillion brianmcgillion brianmcgillion approved these changes

+1 more reviewer

@everton-dematos everton-dematos everton-dematos approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

Tested on Lenovo X1 Carbon This PR has been tested on Lenovo X1 Carbon Tested on System76

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@juliuskoskela @brianmcgillion @milva-unikie @everton-dematos