When using the S3 Asset Store import in Girder being able to import the annotations along with the video/image sequence.

For image-sequence it should automatically be in a folder already and should hit the postprocess endpoint to ingest the CSV/JSON file during the import process.

For Video we need to match up the JSON/csv file with the same name as the video file and move that to the folder as well. It should then move that to a newly created folder and use that while hitting postprocess on it.

The Pub/Sub Google Cloud notifications for buckets will also need to be updated so that it can import updated annotations when they are added as well.

Current System:

There exists an event for processing assetStore imports that will process the data based on extensions and trying to find image sequences as well as video files: https://github.com/Kitware/dive/blob/main/server/dive_server/event.py#L43
This needs to be updated to include looking for a JSON or CSV file with the same exact name and placing it inside of the newly created folder. There is already a ending call to convert_video_recursive (https://github.com/Kitware/dive/blob/main/server/dive_server/event.py#L132) that should call postprocess on all of the new datasets. This should either transcode or import CSV/JSON files that are in the same folder.

Pub/Sub System:

https://kitware.github.io/dive/Deployment-Storage/#pubsub-notifications

The Pub/Sub system is used to notify the deployment of new updates to the GCS bucket. It should import new data or update data as it is added to the bucket. This would need to update so that it will also import and add annotations as those files are added into the system. This would be done at this location in the code:https://github.com/Kitware/dive/blob/main/server/bucket_notifications/views.py#L74

TODO:

• Update the Ingestion process so that it will take in csv/json files that have the same name as the video and import them properly if they exist by placing them in the same newly created folder.
• Test this new system using MinIO
• Test the GCS pub/sub notification system.
• Make sure the new system will support updated annotations as well.