Skip to content

fuse: Add cache for blob sizes#998

Merged
fd0 merged 4 commits intomasterfrom
fix-820
Jun 8, 2017
Merged

fuse: Add cache for blob sizes#998
fd0 merged 4 commits intomasterfrom
fix-820

Conversation

@fd0
Copy link
Copy Markdown
Member

@fd0 fd0 commented Jun 5, 2017

Closes: #820

@fd0 fd0 force-pushed the fix-820 branch 2 times, most recently from 85aa1c0 to 87bfa45 Compare June 6, 2017 19:15
@codecov
Copy link
Copy Markdown

codecov bot commented Jun 6, 2017

Codecov Report

Merging #998 into master will decrease coverage by 0.03%.
The diff coverage is n/a.

Impacted file tree graph

@@            Coverage Diff             @@
##           master     #998      +/-   ##
==========================================
- Coverage   53.91%   53.88%   -0.04%     
==========================================
  Files         116      114       -2     
  Lines       11000    11011      +11     
==========================================
+ Hits         5931     5933       +2     
- Misses       4391     4401      +10     
+ Partials      678      677       -1
Impacted Files Coverage Δ
restic/archiver/archiver.go 64.84% <0%> (-0.17%) ⬇️
restic/backend/s3/s3.go 70.3% <0%> (-0.12%) ⬇️
restic/fuse/dir.go 0% <0%> (ø) ⬆️
cmds/restic/background.go
restic/node_darwin.go
restic/fuse/file.go 68.1% <0%> (+0.55%) ⬆️
restic/backend/test/tests.go 58.82% <0%> (+0.67%) ⬆️
restic/fuse/snapshot.go 3.73% <0%> (+3.73%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 550e1fe...a46baf7. Read the comment docs.

@middelink
Copy link
Copy Markdown
Member

Should we run the tests with -race? maps aren't thread safe and I'm not 100% sure we cannot have multiple file listings in progress. aka, callbacks coming in from fuse.

Side note, this should be abstracted out, no? We now have multiple backends and cmd's where we are adding some form of caching to make things perform. Seems to me we should do this more central. For exmaple, I can imagine this commit not doing much on S3, as it already has a size cache...

@fd0
Copy link
Copy Markdown
Member Author

fd0 commented Jun 7, 2017

Good points, but both aren't relevant here:

  • Maps aren't thread safe, and multiple request from fuse can come in in parallel, that's both true. However, concurrent read from maps is thread safe, and we'll only create the size cache once (when the snapshots dir is constucted), and then it's read only access. So that's not an issue here.
  • The map caches sizes for all blobs in the repo, the data is extracted from the index. The size cache for the s3 backend caches the size of files. It's a different level of abstraction. By the way: The size cache for the s3 backend is not used any more, and in one of the outstanding PRs I've already removed it completely.

I have plans for reworking the index data structures (also to address #979), maybe we can drop this particular cache then.

@fd0
Copy link
Copy Markdown
Member Author

fd0 commented Jun 7, 2017

I've wrapped the cache in a struct and added a Lookup() method to make it clearer that the map is only queried, not changed concurrently.

fd0 added 2 commits June 7, 2017 20:04
Wrap it in a struct and add a Lookup() function to make clear that it
is only queried, not changed, so we don't have any race conditions.
@codecov-io
Copy link
Copy Markdown

Codecov Report

Merging #998 into master will decrease coverage by 0.05%.
The diff coverage is n/a.

Impacted file tree graph

@@            Coverage Diff             @@
##           master     #998      +/-   ##
==========================================
- Coverage   53.91%   53.86%   -0.06%     
==========================================
  Files         116      116              
  Lines       11000    11019      +19     
==========================================
+ Hits         5931     5935       +4     
- Misses       4391     4406      +15     
  Partials      678      678
Impacted Files Coverage Δ
restic/archiver/archiver.go 64.84% <0%> (-0.17%) ⬇️
restic/backend/s3/s3.go 70.3% <0%> (-0.12%) ⬇️
restic/fuse/dir.go 0% <0%> (ø) ⬆️
restic/fuse/file.go 68.1% <0%> (+0.55%) ⬆️
restic/fuse/snapshot.go 3.73% <0%> (+3.73%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 550e1fe...a46baf7. Read the comment docs.

@fd0 fd0 merged commit a46baf7 into master Jun 8, 2017
fd0 added a commit that referenced this pull request Jun 8, 2017
fuse: Add cache for blob sizes
@fd0 fd0 deleted the fix-820 branch June 8, 2017 18:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants