Skip to content

tq: simplify usage of tq.Transfer#1772

Merged
ttaylorr merged 5 commits intotq-masterfrom
tq-object
Dec 15, 2016
Merged

tq: simplify usage of tq.Transfer#1772
ttaylorr merged 5 commits intotq-masterfrom
tq-object

Conversation

@ttaylorr
Copy link
Contributor

This pull-request simplifies the tq.Transfer type, and removes the usage of the mutable Transferable interface. Additionally, it enables us to remove *api.ObjectResource from the tq's exported API.

Previously, we used three main types:

  1. Transferable - an interface{} to cover downloadable and uploadable LFS pointers.
  2. *api.ObjectResource - A type to hold values from the API.
  3. *transfer.Transfer - A combination of Transferable() and *api.ObjectResource. Used in adapter implementations to grab values both from the API, and values pertaining to the local repository.

The bummer is that the Transferable is cached in a map[string]Transferable on the *TransferQueue for the entire transfer, which can incur some pretty high memory consumption. unsafe.Sizeof suggests that the we're storing much more than 56 bytes per Transfer, not including the size of the inner-structs. My calculations estimate about ~128bytes per object (for two actions (32 bytes/action), 4 bytes for the pointer), and no error).

To simplify, let's store the API data when we need it, and ignore it when we don't. For a local cache, all we need is the Name, Path, Oid and Size, which compacts to 56 bytes as a maximum (we can remove 16 bytes/transfer by dropping the Name field). This is, on average, 68.7% less memory consumption per unique Transfer.

In addition, the code gets simpler, since we only need to care about the objects retuned back from the API when we're sending them to the adapters.

On top of all of that, we got rid of the mutability in SetObject(), narrowed down our internal use of the *api.ObjectResource type, and only have one remaining external use of the api package in tq.


Since this PR is on the larger side, here are the important groups of commits:

  1. e18044f: remove Transferable interface
  2. 7f26137: simplify (q *TransferQueue) Add() signature.
  3. 94b2d0d...5fc2344: various 💅 and 🍬 to clean things up

/cc @git-lfs/core

@ttaylorr ttaylorr added this to the v2.0.0 milestone Dec 14, 2016
@ttaylorr
Copy link
Contributor Author

Tracking: #1764.

@ttaylorr ttaylorr mentioned this pull request Dec 14, 2016
5 tasks
Copy link
Contributor

@technoweenie technoweenie left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great! I made a note of a few things in #1764 (comment), but those should probably be handled in separate pull requests. Also, there are a few commented out errors in favor of ActionMissingError or ActionExpiredErr. Those should be removed, but could easily be done elsewhere too. I think we can ship this PR now.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants