Skip to content

audit: add Nix-specific rules#1344

Merged
brianmcgillion merged 1 commit intotiiuae:mainfrom
everton-dematos:pr_audit_logging
Sep 4, 2025
Merged

audit: add Nix-specific rules#1344
brianmcgillion merged 1 commit intotiiuae:mainfrom
everton-dematos:pr_audit_logging

Conversation

@everton-dematos
Copy link
Copy Markdown
Contributor

@everton-dematos everton-dematos commented Aug 29, 2025 •
edited
Loading

Description of Changes

Enhances NixOS audit coverage, aligns with Anduril NixOS STIG guidance, removes overlapping/erroring rules, and adds opt-in verbose toggles.

Key Updates

  • Nix-specific coverage
    • Exec rules for nix-daemon and nix (execve/execveat).
    • Path watches for nix, nix-store, nix-shell, nix-build, nix-collect-garbage, and nixos-rebuild.
    • Watches for /etc/nix, /etc/nixos, and nix-daemon unit/socket paths.
    • Host-level watches for Nix profiles, DB/state, GC lock, and system generation symlinks.
  • STIG alignment
  • Verbose options
    • ghaf.security.audit.enableVerboseCommon
    • ghaf.security.audit.enableVerboseOspp

Performance Insights

Storage

  • Write rate (KiB/s) is measured by monitoring the audit.log files.
    • Each scenario was run 3×. The following table presents the average of the three runs with different configurations (i.e., scenarios) for two phases: (i) booting, (ii) idle (~10 min)
Scenario Common rules Common-Verbose OSPP OSPP-Verbose STIG #Rules Boot avg (KiB/s) Idle avg (KiB/s)
net-vm-S1 ✅ ❌ ❌ ❌ ❌ 43 5.7757 0.0742
net-vm-S2 ✅ ✅ ❌ ❌ ❌ 45 138.1898 1.3372
net-vm-S3 ✅ ❌ ✅ ❌ ❌ 82 5.7824 0.1698
net-vm-S4 ✅ ❌ ❌ ✅ ❌ 83 21.0952 0.1552
net-vm-S5 ✅ ❌ ❌ ❌ ✅ 60 6.4002 0.107
host-vm-S1 ✅ ❌ ❌ ❌ ❌ 47 2.4155 0.2051
host-vm-S2 ✅ ✅ ❌ ❌ ❌ 49 116.2952 2.3102
host-vm-S3 ✅ ❌ ✅ ❌ ❌ 86 2.1979 0.2053
host-vm-S4 ✅ ❌ ❌ ✅ ❌ 87 2.6081 0.2065
host-vm-S5 ✅ ❌ ❌ ❌ ✅ 64 2.8714 0.2053

Memory and CPU

  • Measured on auditd.service during idle (~10 min).
    • Memory: average of systemd cgroup MemoryCurrent and peak from MemoryPeak.
    • CPU: average % over the same window, computed from deltas of the main PID’s utime+stime (i.e., cpu_pct).
Scenario Common rules Common-Verbose OSPP OSPP-Verbose STIG #Rules Memory avg (MiB) Memory peak (MiB) CPU (%)
net-vm-S1 ✅ ❌ ❌ ❌ ❌ 43 0.76 3.66 0.01
net-vm-S2 ✅ ✅ ❌ ❌ ❌ 45 3.20 5.65 0.02
net-vm-S3 ✅ ❌ ✅ ❌ ❌ 82 0.89 3.18 0.01
net-vm-S4 ✅ ❌ ❌ ✅ ❌ 83 5.92 9.34 0.19
net-vm-S5 ✅ ❌ ❌ ❌ ✅ 60 0.77 3.36 0.01
host-vm-S1 ✅ ❌ ❌ ❌ ❌ 47 1.96 3.15 0.01
host-vm-S2 ✅ ✅ ❌ ❌ ❌ 49 12.62 13.46 0.02
host-vm-S3 ✅ ❌ ✅ ❌ ❌ 86 2.16 2.65 0.01
host-vm-S4 ✅ ❌ ❌ ✅ ❌ 87 10.23 13.85 0.11
host-vm-S5 ✅ ❌ ❌ ❌ ✅ 64 1.99 2.42 0.01

Type of Change

  • New Feature
  • Bug Fix
  • Improvement / Refactor

Related Issues / Tickets

Checklist

  • Clear summary in PR description
  • Detailed and meaningful commit message(s)
  • Commits are logically organized and squashed if appropriate
  • Contribution guidelines followed
  • Ghaf documentation updated with the commit - https://tiiuae.github.io/ghaf/
  • Author has run make-checks and it passes
  • All automatic GitHub Action checks pass - see actions
  • Author has added reviewers and removed PR draft status

Testing Instructions

Applicable Targets

  • Orin AGX aarch64
  • Orin NX aarch64
  • Lenovo X1 x86_64
  • Dell Latitude x86_64
  • System 76 x86_64

Installation Method

  • Requires full re-installation
  • Can be updated with nixos-rebuild ... switch
  • Other:

Test Steps To Verify:

  1. To enable audit you can use either the extras (#lenovo-x1-extras-debug) - audit will be enabled by default - or set this flag to true (https://github.com/everton-dematos/ghaf/blob/pr_audit_logging/modules/reference/profiles/mvp-user-trial.nix#L102)
  2. The rules will be available in all the VMs. The ghaf-host will have a few more rules, as it has specific rules for host (https://github.com/everton-dematos/ghaf/blob/pr_audit_logging/modules/common/security/audit/rules/host.nix)
  3. To enable specific set of rules (i.e., enableVerboseCommon, enableStig, enableOspp, enableVerboseOspp), it is mandatory to set it as true, as follows:
# Optional hardening:
ghaf.security.audit.enableStig = true;
ghaf.security.audit.enableOspp = true;

# Noisy diagnostics (opt-in):
ghaf.security.audit.enableVerboseCommon = true;
ghaf.security.audit.enableVerboseOspp = true;
  1. After boot, verify that auditd.service and audit-rules.service are up and running by systemctl status <service>
  2. List the loaded rules by sudo auditctl -l
  3. Monitor /var/log/audit/ files

@everton-dematos everton-dematos requested a review from mbssrc August 29, 2025 11:51
@brianmcgillion brianmcgillion added the Needs Testing CI Team to pre-verify label Sep 2, 2025
mbssrc
mbssrc reviewed Sep 2, 2025
View reviewed changes
@leivos-unikie
Copy link
Copy Markdown
Contributor

leivos-unikie commented Sep 3, 2025

Tested on Lenovo-X1

  • Set
    security.audit.enable = true;
    in
    ghaf/modules/reference/profiles/mvp-user-trial.nix
    and build
    nix build .#lenovo-x1-carbon-gen11-debug

  • Boot from USB SSD

  • Checked
    auditd.service
    audit-rules.service
    sudo auditctl -l
    /var/log/audit/audit.log
    in every VM

  • All is good except zathura-vm failed audit-rules.service

  • zathura-vm has only single line in audit.log
    [root@zathura-vm:/var/log/audit]# cat audit.log
    node=zathura-vm type=DAEMON_START msg=audit(1756901748.310:3519): op=start ver=4.1.0 format=raw kernel=6.12.41 auid=4294967295 pid=374 uid=0 ses=4294967295 res=success

  • zathura-vm has "No rules"

  • In zathura-vm there was also something else strange:
    journalctl didn't contain logs from the first boot although other vms have also history of the previous boots in their journactl (saved in some persistent storage)

There is a bug about prolonged start of zathura-vm and gala-vm. I suspect that it might be related.
https://jira.tii.ae/browse/SSRCSP-6918

Captured journalctl logs:
ghaf-host_journalctl.log
zathura_journalctl.log

@leivos-unikie leivos-unikie added bug on Lenovo X1 Carbon Issues found on Lenovo X1 Carbon while checking this PR and removed Needs Testing CI Team to pre-verify labels Sep 4, 2025
@everton-dematos
audit: add Nix-specific rules
0d5455a 
Signed-off-by: Everton de Matos <everton.dematos@tii.ae>
@everton-dematos
Copy link
Copy Markdown
Contributor Author

everton-dematos commented Sep 4, 2025

Tested on Lenovo-X1

* Set
  `security.audit.enable = true;`
  in
  `ghaf/modules/reference/profiles/mvp-user-trial.nix`
  and build
  `nix build .#lenovo-x1-carbon-gen11-debug`

* Boot from USB SSD

* Checked
  `auditd.service`
  `audit-rules.service`
  `sudo auditctl -l`
  `/var/log/audit/audit.log`
  in every VM

* All is good except zathura-vm failed audit-rules.service

* zathura-vm has only single line in audit.log
  [root@zathura-vm:/var/log/audit]# cat audit.log
  node=zathura-vm type=DAEMON_START msg=audit(1756901748.310:3519): op=start ver=4.1.0 format=raw kernel=6.12.41 auid=4294967295 pid=374 uid=0 ses=4294967295 res=success

* zathura-vm has "No rules"

* In zathura-vm there was also something else strange:
  journalctl didn't contain logs from the first boot although other vms have also history of the previous boots in their journactl (saved in some persistent storage)

There is a bug about prolonged start of zathura-vm and gala-vm. I suspect that it might be related. https://jira.tii.ae/browse/SSRCSP-6918

Captured journalctl logs: ghaf-host_journalctl.log zathura_journalctl.log

Fix applied. I updated modules/reference/appvms/zathura.nix to adjust the audit-rules.service start order so it runs in the normal boot phase instead of early-boot.

Regarding logs: the missing prior-boot logs on zathura-vm are intentional. This VM is configured to be stateless (ghaf.storagevm.enable = false), so its logs are not persisted across boots by design.

Also rebased the branch.

@leivos-unikie leivos-unikie removed the bug on Lenovo X1 Carbon Issues found on Lenovo X1 Carbon while checking this PR label Sep 4, 2025
@leivos-unikie
Copy link
Copy Markdown
Contributor

leivos-unikie commented Sep 4, 2025

Checked after the fix that now the audit feature works in every VM as described.

Status of audit-rules.service is "active (exited)" in every vm but I guess the service is supposed to run only at boot and it is as it should be.

@leivos-unikie leivos-unikie added the Tested on Lenovo X1 Carbon This PR has been tested on Lenovo X1 Carbon label Sep 4, 2025
@brianmcgillion brianmcgillion merged commit c82c2f9 into tiiuae:main Sep 4, 2025
28 checks passed
@everton-dematos everton-dematos deleted the pr_audit_logging branch January 23, 2026 07:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@mbssrc mbssrc mbssrc approved these changes

@vunnyso vunnyso vunnyso approved these changes

Assignees

No one assigned

Labels

Tested on Lenovo X1 Carbon This PR has been tested on Lenovo X1 Carbon

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@everton-dematos @leivos-unikie @mbssrc @vunnyso @brianmcgillion