Skip to content

Refactor ipfs get to use CoreAPI#5943

Merged
Stebalien merged 5 commits intomasterfrom
misc/coreapi-get
Jan 30, 2019
Merged

Refactor ipfs get to use CoreAPI#5943
Stebalien merged 5 commits intomasterfrom
misc/coreapi-get

Conversation

@magik6k
Copy link
Copy Markdown
Member

@magik6k magik6k commented Jan 24, 2019 •
edited by Stebalien
Loading

@magik6k magik6k added the topic/core-api Topic core-api label Jan 24, 2019
@magik6k magik6k requested a review from Kubuxu as a code owner January 24, 2019 19:40
@ghost ghost assigned magik6k Jan 24, 2019
@ghost ghost added the status/in-progress In progress label Jan 24, 2019
This was referenced Jan 24, 2019
Open
Closed
hannahhoward
hannahhoward approved these changes Jan 29, 2019
View reviewed changes
Copy link
Copy Markdown
Contributor

@hannahhoward hannahhoward left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM and on the the two connected PRs.. but I don't consider myself a core API expert by any means.

I have two quick questions:

  1. Why move unixfile to go-unixfs & why put tar-writer in go-ipfs-files instead of go-unixfs?
  2. Am I correct in understanding that neither of these two changes are strickly essential to delivering the refactor to make get use CoreAPI?

Non-blocking, just want to understand.

@Stebalien
Copy link
Copy Markdown
Member

Stebalien commented Jan 29, 2019

why put tar-writer in go-ipfs-files

Because coreapi.Unixfs().Get() now returns a go-ipfs-files.Node object. Therefore, the TarWriter needs to convert this (instead of a dag) into a tarfile.

However: ipfs/go-unixfs#59 (comment)

Am I correct in understanding that neither of these two changes are strickly essential to delivering the refactor to make get use CoreAPI?

We need some way to go from a go-ipfs-files.Node to a tar because coreapi.Unixfs().Get() returns a go-ipfs-files.Node instead of a raw IPLD DAG. We could just keep DagArchive as it is as a way to go directly from a DAG to a tar file.

@magik6k
Copy link
Copy Markdown
Member Author

magik6k commented Jan 30, 2019

We could just keep DagArchive as it is as a way to go directly from a DAG to a tar file.

As a note - I decided to drop it as the combo of NewUnixfsFile+files.TarWriter do the same thing with the only difference being the files interface in between (there shouldn't be any meaningful performance impact). Plus after switching ipfs get to the new thing, it would basically become dead code.

This was referenced Jan 30, 2019
Merged
Merged
@magik6k
coreapi: support symlinks in unixfile
e4020f9 
License: MIT
Signed-off-by: Łukasz Magiera <magik6k@gmail.com>
@magik6k
Use coreapi in ipfs get
aa3e087 
License: MIT
Signed-off-by: Łukasz Magiera <magik6k@gmail.com>
@magik6k
coreapi: extract unixfile to go-unixfs
a151041 
License: MIT
Signed-off-by: Łukasz Magiera <magik6k@gmail.com>
@magik6k
commands/get: move FileArchive here
e71e700 
License: MIT
Signed-off-by: Łukasz Magiera <magik6k@gmail.com>
Stebalien
Stebalien approved these changes Jan 30, 2019
View reviewed changes
core/commands/get.go
return err
}

res.SetLength(uint64(size))
Copy link
Copy Markdown
Member

@Stebalien Stebalien Jan 30, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is actually incorrect but it's an existing bug... #5690

Copy link
Copy Markdown
Member Author

@magik6k magik6k Jan 30, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The alternative is not setting this at all, but it will break progress bar... (We could pass this in a separate header)

Copy link
Copy Markdown
Member

@Stebalien Stebalien Jan 30, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we have to. Unfortunately, that'll require a change to the commands library.

Stebalien
Stebalien requested changes Jan 30, 2019
View reviewed changes
Copy link
Copy Markdown
Member

@Stebalien Stebalien left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like this doesn't work with sharding. t0260-sharding.sh is failing.

@Stebalien
Copy link
Copy Markdown
Member

Stebalien commented Jan 30, 2019

Error is: "Error: cannot write to readonly DAGService"

@Stebalien
Copy link
Copy Markdown
Member

Stebalien commented Jan 30, 2019

The issue is: https://github.com/ipfs/go-unixfs/blob/0eb1ef8f33371b1b9f74f74655b584144919ce91/file/unixfile.go#L121. This is causing the hamt to serialize and save itself.

@magik6k
Copy link
Copy Markdown
Member Author

magik6k commented Jan 30, 2019

It should probably just do return 0, files.ErrNotSupported, I'll try to see how it affects things

@magik6k
Copy link
Copy Markdown
Member Author

magik6k commented Jan 30, 2019

Ah, right, it breaks the progress bar ._.

@Stebalien
Copy link
Copy Markdown
Member

Stebalien commented Jan 30, 2019

Maybe we should have a separate hamt reader?

@magik6k magik6k mentioned this pull request Jan 30, 2019
Merged
@magik6k
Copy link
Copy Markdown
Member Author

magik6k commented Jan 30, 2019

See ipfs/go-unixfs#61

@magik6k
gx: update go-unixfs to propagate archive changes
7c2aa0e 
License: MIT
Signed-off-by: Łukasz Magiera <magik6k@gmail.com>
@magik6k magik6k mentioned this pull request Jan 30, 2019
Merged
@magik6k magik6k requested a review from Stebalien January 30, 2019 20:30
@Stebalien Stebalien merged commit 35aaa98 into master Jan 30, 2019
@Stebalien Stebalien deleted the misc/coreapi-get branch January 30, 2019 20:52
@ghost ghost removed the status/in-progress In progress label Jan 30, 2019
hacdias pushed a commit to ipfs/boxo that referenced this pull request Jan 27, 2023
@Stebalien
Merge pull request ipfs/kubo#5943 from ipfs/misc/coreapi-get
573067d 
Refactor ipfs get to use CoreAPI

This commit was moved from ipfs/kubo@35aaa98
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@Stebalien Stebalien Stebalien approved these changes

@hannahhoward hannahhoward hannahhoward approved these changes

@Kubuxu Kubuxu Awaiting requested review from Kubuxu

Assignees

@magik6k magik6k

Labels

topic/core-api Topic core-api

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@magik6k @Stebalien @hannahhoward @eingenito