Tested the PR on lenovo-x1-carbon-gen10 machine .
{machine="<machine_id>"} |~ "PATH" |~ "/nix/store/"
{machine="<machine_id>"} |= "nixos_rebuild_store"
shows actual nixos_rebuild_store logs with hash ,timestamp and origin.

However the failed nixos-rebuild attempts are not captured quite well according to the original requirement https://jira.tii.ae/browse/SSRCSP-6933
created a bug for the same https://jira.tii.ae/browse/SSRCSP-7486

I was curious to see if that write-watch is able to log also direct write to /nix/store after bypassing read-only mode by remount.

[root@ghaf-host:/var/empty]# mount -l -o remount,rw,relatime overlay /nix/store
[root@ghaf-host:/var/empty]# cd /etc/static
[root@ghaf-host:/etc/static]# ls -la | grep fstab
[root@ghaf-host:/etc/static]# echo "# can you see this" >> /nix/store/dym70yrx7kdyl0wqh3i6kjvi47f634wh-etc-fstab

write-watch was able to see that write into nix store, grafana logged:

SYSCALL arch=c000003e syscall=257 success=yes exit=3 a0=ffffff9c a1=5595cfb45260 a2=441 a3=1b6 items=1 ppid=7288 pid=7290 auid=1001 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts1 ses=5 comm="bash" exe="/nix/store/2j7r5np0vaz4cnqkymp1mqivmjj1x9xl-bash-interactive-5.3p3/bin/bash" key="nixos_rebuild_store"

PATH item=0 name="/nix/store/dym70yrx7kdyl0wqh3i6kjvi47f634wh-etc-fstab" inode=2157377 dev=08:04 mode=0100444 ouid=0 ogid=0 rdev=00:00 nametype=NORMAL cap_fp=0 cap_fi=0 cap_fe=0 cap_fver=0 cap_frootid=0

I was curious to see if that write-watch is able to log also direct write to /nix/store after bypassing read-only mode by remount.

[root@ghaf-host:/var/empty]# mount -l -o remount,rw,relatime overlay /nix/store
[root@ghaf-host:/var/empty]# cd /etc/static
[root@ghaf-host:/etc/static]# ls -la | grep fstab
[root@ghaf-host:/etc/static]# echo "# can you see this" >> /nix/store/dym70yrx7kdyl0wqh3i6kjvi47f634wh-etc-fstab

write-watch was able to see that write into nix store, grafana logged:

SYSCALL arch=c000003e syscall=257 success=yes exit=3 a0=ffffff9c a1=5595cfb45260 a2=441 a3=1b6 items=1 ppid=7288 pid=7290 auid=1001 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts1 ses=5 comm="bash" exe="/nix/store/2j7r5np0vaz4cnqkymp1mqivmjj1x9xl-bash-interactive-5.3p3/bin/bash" key="nixos_rebuild_store"

PATH item=0 name="/nix/store/dym70yrx7kdyl0wqh3i6kjvi47f634wh-etc-fstab" inode=2157377 dev=08:04 mode=0100444 ouid=0 ogid=0 rdev=00:00 nametype=NORMAL cap_fp=0 cap_fi=0 cap_fe=0 cap_fver=0 cap_frootid=0

If I manage to edit a symbolic link pointing to file in /nix/store
[root@ghaf-host:/etc/static]# echo "# can you see this also" >> fstab
exact target path of the symbolic link is not logged in grafana but /nix/store and fstab are mentioned

CWD cwd="/nix/store"
PATH item=0 name="fstab" inode=2167558 dev=08:04 mode=0100644 ouid=0 ogid=0 rdev=00:00 nametype=NORMAL cap_fp=0 cap_fi=0 cap_fe=0 cap_fver=0 cap_frootid=0

Tested the PR on lenovo-x1-carbon-gen10 machine . {machine="<machine_id>"} |~ "PATH" |~ "/nix/store/" {machine="<machine_id>"} |= "nixos_rebuild_store" shows actual nixos_rebuild_store logs with hash ,timestamp and origin.

However the failed nixos-rebuild attempts are not captured quite well according to the original requirement https://jira.tii.ae/browse/SSRCSP-6933 created a bug for the same https://jira.tii.ae/browse/SSRCSP-7486

I added a "Store Watcher" service to monitor the nix store - https://github.com/everton-dematos/ghaf/blob/pr_logging_nixos_rebuild/modules/common/services/storewatcher.nix

It is a bit tricky to monitor failed nixos-rebuild operations, as, from the target side, it involves an SSH connection with a switch-to-configuration command at the end. Therefore, it is easier to monitor successful attempts than failed ones.

What I did with this "Store Watcher" is to monitor the modifications at the nix store location and trigger a message if a transfer is taking too long. I added cycles of "checking", as in some cases, the transfer of a specific package may take a while, either because the package is large in size or due to network issues. The number of cycles of waiting and the time for each cycle can be personalized by parameters.

Testing:

• To enable it, we have to set storeWatcher.enable = true; at microvm-host.nix - It can be in line 101, just after this line: https://github.com/everton-dematos/ghaf/blob/pr_logging_nixos_rebuild/modules/microvm/host/microvm-host.nix#L100

• To check if the service is working fine, please run systemctl status nixos-rebuild-watch.service at ghaf@ghaf-host

• Whenever you start a nixos-rebuild from another machine, you will be able to see the logs of the target at Grafana:

  • {machine="<machine_id>"} |= store-copy-session - will show all the packages transferred, and if the transfer was likely aborted. You should expect to see something like this:

    • 

  • {machine="machine_id"} |= store-copy-debug- will show a more verbose version of the logs. You should expect to see something like this:

    • 

  • In case of a successful nixos-rebuild, the log will print journal store-copy-session ok. You should expect to see something like this:

    • 

• Important: If you start a nixos-rebuild and then stop it during the copy of the files, it will take some time for the likely aborted log to appear in Grafana, as it will wait for some cycles before printing the likely aborted message. Currently, with the defined parameters, it will wait for 5 cycles, each lasting 60 seconds. However, you can see the first log immediately after the package transfer is initiated, and the subsequent "cycle" logs.

Performance:

• The following shows the performance metrics while idle - no nixos-rebuild happening:
  • CPU %:
    • Average = 0.00
    • Peak = 0.00
  • Memory MiB:
    • Average = 4.14
    • Peak = 4.14
    • 
• The following shows the performance metrics during a nixos-rebuild execution:
  • CPU %:
    • Average = 4.08
    • Peak = 29.31
  • Memory MiB:
    • Average = 3.00
    • Peak = 5.87
  •