@whyrusleeping

From an inline comment above:

I don't know what you have in mind for the filestore. But I very specially designed my filestore so that writes go though. This way if a file is moved it can simply be readded and any invalid blocks are automatically fixed.

That becomes problematic, What then happens if i add dataset A, which contains some subset C. Then later, i add dataset B that also contains some subset C. Then i change my mind and remove dataset B. By your description, the original dataset references would now be lost, with no indication to the user that this has happened.

Yes this was originally a problem. In my implementation I fixed this. See ipfs-filestore#12 and ipfs-filestore#23. However, that solution also requires writes to go through.

I think this scenario is worse UX than the reverse (deleting A causing bad refs in B). In either case, repairs are going to have to be manual, and some things will get sticky.

Not if my solution in ipfs-filestore#12 is implemented.

cc @jbenet for his thoughts on this scenario.

Yes this was originally a problem. In my implementation I fixed this. See ipfs-filestore#12 and ipfs-filestore#23. However, that solution also requires writes to go through.

Another option is to only replace the block if it is invalid. I considered that, but decided that it was better to support allowing multiple files to point to the same block.

I see that "--nocopy" is allowed when the daemon is online without any sort of additional checks. If the blocks are ever not verified this could become a security problem. Even without the security problems this could lead to strange results if files are added on another machine with an identical path.

For this to be usable outside of ipfs-pack I would strongly recommend that references be allowed "outside the root handler" by simply requiring all paths to be absolute.

@whyrusleeping concerning replacing blocks. How about for now we make it an option? There needs to be a way to fix broken blocks due to files moving. One option would be to remove the block and then readd it, but pinning makes that a more complicated operation.

As far as allowing multiple blocks for a hash (ipfs-filestore#12 and ipfs-filestore#23), I agree that we should aim for that but to start with it might be a bit much to review.

Is this branch ready for me to test on my 30 TB of webcam videos or should I wait?

@jefft0 Its almost ready, should be merged in a couple days.

Rebased, waiting on a 👍 from @Kubuxu and CI

At @jefft0 I am really not sure how usable this code is going to be for you as it does not allow for files "outside the root handler". For example if you directory layout is:

/home/jeff0/.ipfs
/aux/media

and you want to share files in /aux/media you won't be able to and will get an error.

@whyrusleeping I would like this comment, to at least be addressed:

I see that "--nocopy" is allowed when the daemon is online without any sort of additional checks. If the blocks are ever not verified this could become a security problem. Even without the security problems this could lead to strange results if files are added on another machine with an identical path.

@kevina @jefft0 In this case, simply set your IPFS_PATH to be in (given the example) /aux/ipfs or similar, We're pushing for git style security on file references. They need to be within the scope of the ipfs repository.

@kevina Re the comment, blocks are checked now (thanks for pointing that out), and this feature should not be used with client and daemon on separate machines. We can probably implement a check of some sort for this... any ideas on how to do that off the top of your head?

I'm confused. I have several webcam videos totalling 60 GB. With a fresh installation of go-ipfs (master branch), I did ipfs init then ipfs add --nocopy --raw-leaves for each file. But now ~/.ipfs is 60 GB. Is --nocopy working yet in master?

@whyrusleeping anything on this?

Ah, theres a check i forgot to add, erroring out if --nocopy is passed without enabling the filestore.

@jefft0 take a look at "How to enable" here: #3397 (comment)

@whyrusleeping Much better, thanks! Now ~/,ipfs is only 68 MB. I'll try adding a lot more files. Where do you want me to report my results?

@jefft0 You can report them here, or open a new issue to discuss. Either way :)

@whyrusleeping I want to keep my main repo in ~/.ipfs, so instead of changing IPSF_PATH I put a symbolic link to my external drive in ~/.ipfs and do ipfs add --nocopy with that path. This works. It would also let me use multiple external drives at different mount points. Do you see any downside to this approach?

@jefft0 hrm... unsure about the implications of this. I'll have to think about the security model of using/allowing symlinks like that.

My first thought though is that it should be fine

@jefft0 you can also put the symbolic link in just ~/, it doesnt have to be IN the ipfs dir, just in its parent (like the .git folder for git repos)

every time I get

$ ps aux | grep ipfs
atommixz  3109 36.2  2.5 879840 207656 ?       Ssl  23:20   1:24 /usr/bin/ipfs daemon --manage-fdlimit=true

$ LANG=C type ipfs_add
ipfs_add is aliased to `ipfs add --nocopy --raw-leaves --recursive --progress'

$ ipfs_add download/torrents
added QmcGUHBF2XT8dTblablablazYsNQch6ECGx torrents/file.ext
 3.24 GB / 61.22 GB [==>---------------------------------------------------]   5.30% 15m33s23:23:30.659 ERROR commands/h: open /home/username/.ipfs/blocks/HE/put-789802368: too many open files client.go:247
Error: open /home/username/.ipfs/blocks/HE/put-789802368: too many open files

@atommixz Wanna open a new issue for this?

This helps to me:

$ systemctl --user edit --full ipfs 
[Service]
LimitNOFILE=999999

This is don't make sence to me ipfs daemon --manage-fdlimit=true

I'm trying to download many images from filestore. Everytime it stops at the same place. Last commit on two pc.

$ ipfs get Qmblablabla
Saving file(s) to Qmblablabla
 117.00 MB / 2.36 GB [==>------------------------------------------------------]   4.84% 4s
Error: expected protobuf dag node

...
23:49:27.028 ERROR    bitswap: couldnt open sender again after SendMsg(<peer.ID PRB9SN>) failed: dial attempt failed: <peer.ID Rj75eS> --> <peer.ID PRB9SN> dial attempt failed: i/o timeout wantmanager.go:233
23:50:03.062 ERROR    bitswap: couldnt open sender again after SendMsg(<peer.ID aoqJoJ>) failed: dial attempt failed: <peer.ID Rj75eS> --> <peer.ID aoqJoJ> dial attempt failed: i/o timeout wantmanager.go:233
23:51:15.626 ERROR commands/h: err: expected protobuf dag node handler.go:288
23:52:34.252 ERROR commands/h: err: expected protobuf dag node handler.go:288
23:52:59.859 ERROR commands/h: err: expected protobuf dag node handler.go:288

Can you report it in separate issue. It is much easier to track this way.

whyrusleeping said "You can report them here, or open a new issue to discuss. Either way :)"
so I do, because it's about filestore.

I think he meant results as it, it worked, performance, small problems.

Yeah, Keeping perf and general 'how it went' things here is fine, but actual problems need to have their own issues so we can address them

I had a problem with too many open files and opened issue #3763. I don't know if it's related to the issue @Kubuxu has.