(See updated proposal below #1958 (comment))

Background

There are situations where Kopia needs to communicate with the user to let them know something important has happened.

Examples:

• data corruption detected
• snapshot failed
• maintenance failed
• an important upgrade is available

Today there's no good way to communicate such things to the user in the UI. Things are even worse in the CLI, since we don't know whether the user is looking at the output at all.

Proposal

1. We introduce the notion of notifications, which will be stored in repository as manifest entries.
2. Notifications will persist in the repository until the user sees them, acknowledges them or until they expire.
3. Notifications can be addressed to: all users or a particular users on the machine.
4. Notifications will be associated with severity: normal, warning, error.
5. We will implement UX to show notifications to users in both the CLI and UI.
6. We will add flags and policy options with useful defaults to control when the notifications are emitted.
7. We should have a goal of not showing notifications to most users unless something really important or catastrophic has happened or if the user explicitly opts in to notifications.
8. In the future, we may also add ability to forward notifications via email, slack, pushover, pagerduty, etc. and via other integrations. This could be specified via user-level configuration or via policies.

CLI behavior changes

At the end of each command execution in the CLI, if there are any pending notifications for the current user, Kopia will display (to stderr) a one-line summary similar to:

$ kopia snapshot create ...
<...>
Created snapshot ... 
NOTICE: There are N pending notifications. To read them use kopia notifications.

Depending on severity, this will say something like.

• There are X pending notifications. To read them use kopia notification list
• There are X pending notifications including Y warnings and Z errors. To read them use kopia notification list

This behavior will be controlled by a flag:

--notifications=hidden | summary | details

• In hidden mode, nothing is displayed
• in summary mode the one-line notice is displayed if there are any notification (red if there are any errors). Also, any warnings or errors will cause exit code 1 to be returned. This will hopefully raise alerts in systems such as crontab and will alert appropriate system administrator who may otherwise be unaware of issues.
• In details mode, all notifications are directly displayed following regular output of a command

This behavior can also be configured using KOPIA_NOTIFICATIONS environment variable.

Listing and acknowledging notifications

$ kopia notification list
2022-05-19 ERROR Maintenance failed on SOME_MACHINE
  To see the details `kopia notification show aaaaaaaaa`
  Go to https://kopia.io/docs/<some-page> for more information on resolving this issue.

2022-05-12 WARNING Snapshot failed on SOME_MACHINE
  To see the details `kopia notification show bbbbbbbbb`

$ kopia notification ack --all | --item= aaaaaaaaa
All pending notifications have been acknowledged.

$ kopia notification list
There are no notifications.

Additionally we may support auto-acking:

$ kopia notification list --auto-ack

UI changes

In the web UI, kopia will have a new tab showing notifications including some alert indicators on the main page. Details TBD.
In the KopiaUI kopia will additionally be showing incoming notifications as desktop popups.

Notification sources

We will add new options to control emitting notifications.

1. Maintenance errors will by default notify the maintenance owner user, but this can be changed with kopia maintenance set flags. --on-error-notify=none | default | username@hostname

2. For kopia snapshot create we will add --on-error-notify=username@hostname

3. We will also add policy options and corresponding flags in kopia policy set to specify notification parameters for each snapshot source.

For example: kopia policy set some-path --on-snapshot-error=notify --notify-user=someuser@somehost

4. Later we will add additional configuration for notifying users about important software upgrades, etc.

Comments welcome, many of the ideas here are just initial thoughs to get the conversation started, not a complete design by any stretch.

If folks are interested in implementing parts of this design or the whole thing, I'll be happy to be help.