Skip to content
This repository was archived by the owner on Sep 12, 2018. It is now read-only.
This repository was archived by the owner on Sep 12, 2018. It is now read-only.

NG: Transactional storage #626

@wking

Description

@wking

We've had some difficulties ensuring atomic transactions with the current registry. This makes it difficult to do things like robustly refcount images (or anything else where you'd need to touch metadata in several locations at once). I've been poking around to get a feel for how other folks handle transactions with huge.

  • Redis (and most key/value stores we might use for the smaller metadata) supports transactions, and there is support in asyncio-redis, if we end up implementing the new registry in Python with aiohttp.
  • Transactional filesystems seem a bit harder to come by, but if you don't have a huge volume of them, you can likely squeak by with something like Toggenburger's libbtrfstrans (although the paper doesn't seem to be maintained or have version-controlled code).
  • Microsoft's SQL Server supports FILESTREAM, which seems to give transactional access through the database layer, but stores the actual blobs transparently to the filesystem. Oracle Database has something similar with BFILE.
  • Gropengießer and Sattler have a paper on transactional S3. I only skimmed it, but it seems to be talking about a unpublished code for their client library and service layer.

This is not my area of expertise though, and its certainly possible that there's a mainstream, open-source, transactional storage solution for large binary blobs; I just can't find it ;).

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions