Description of Changes

This PR addresses cases where log forwarding could stall after journald rotation, resulting in no logs being sent to Grafana.

• Fix journal access after rotation by enforcing persistent journal directory permissions via systemd-tmpfiles and adding required SupplementaryGroups (systemd-journal, adm) to the Alloy service - as peer loki documentation https://grafana.com/docs/alloy/latest/reference/components/loki/loki.source.journal/

• Limit replay of stale journald entries by setting max_age = "168h" on journal sources to avoid forwarding very old logs.

• Add batching and timeout tuning (batch_size, max_backoff_period, remote_timeout) to improve resilience when the remote Loki endpoint is slow or unavailable, helping prevent pipeline stalls.

• Introduce a short older_than = "15m" drop window for specific processing stages to keep ingestion moving under current remote Grafana/Loki constraints.

Type of Change

• New Feature
• Bug Fix
• Improvement / Refactor

Related Issues / Tickets

https://jira.tii.ae/browse/SSRCSP-7612

Checklist

• Clear summary in PR description
• Detailed and meaningful commit message(s)
• Commits are logically organized and squashed if appropriate
• Contribution guidelines followed
• Ghaf documentation updated with the commit - https://tiiuae.github.io/ghaf/
• Author has run make-checks and it passes
• All automatic GitHub Action checks pass - see actions
• Author has added reviewers and removed PR draft status

Testing Instructions

Applicable Targets

• Orin AGX aarch64
• Orin NX aarch64
• Lenovo X1 x86_64
• Dell Latitude x86_64
• System 76 x86_64

Installation Method

• Requires full re-installation
• Can be updated with nixos-rebuild ... switch
• Other:

Test Steps To Verify:

1. Verify that logs are being sent to Grafana
2. This PR should solve the "20 to 30 minutes issue" where logs stopped going to Grafana after rotation - https://jira.tii.ae/browse/SSRCSP-7612