Skip to content

Add support for data change notifications #958

@RomanIakovlev

Description

@RomanIakovlev

The need to enable ClearlyDefined data replication has been discussed already (#386). I'd like to revive this discussion and propose to approach this problem as a 2 step process:

  1. Enable an externally visible notifications about changes in definitions, from both harvesting and curation. An event stream, if you'd like.
  2. Implement data dump of all the existing definitions, with a selected set of attributes, to an external location (e.g. a file, a storage bucket, etc.).

This issue focuses on the first part, enabling the data change notifications. I propose to use a persistent web hook mechanism for this purpose. The list of requirements for ClearlyDefined service to implement this looks roughly like the following:

  • Call an external web service when some data change happens.
  • This call should keep on working after ClearlyDefined service restarts without any actions from the recipient (the persistent part of the definition above).
  • Call URL should support secrets, e.g. auth credentials, without hardcoding them in the ClearlyDefined source code (obviously).
  • There should be support for multiple web hooks, to have multiple events destinations.
  • It is ok to use an ad-hoc, low-level approach for setting up new web hooks, e.g. via a PR to ClearlyDefined service. No web UI is required, at least initially.

Please feel free to add further requirements if they seem necessary, but having this seems to be sufficient for a start.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions