Skip to content

Enable Filecoin retrievals in Buckets#533

Merged
jsign merged 6 commits intomasterfrom
jsign/retb
Mar 22, 2021
Merged

Enable Filecoin retrievals in Buckets#533
jsign merged 6 commits intomasterfrom
jsign/retb

Conversation

@jsign
Copy link
Copy Markdown
Contributor

@jsign jsign commented Mar 18, 2021

This PR sits on top of textileio/powergate#807

We had this feature hidden because of a problem when configuring Lotus when using IpfsOnlineMode=true, which didn't make retrievals possible. The Powergate PR workaround this problem, and unblocks the feature.

This PR also added some extra fixes and sections in the Miner Index CLI.

@jsign jsign self-assigned this Mar 18, 2021
@jsign jsign force-pushed the jsign/retb branch 2 times, most recently from 447629b to dc15891 Compare March 19, 2021 13:14
Comment on lines +469 to +471
if ci.CidInfo.CurrentStorageInfo == nil {
continue
}
Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Archives that are in progress don't have CurrentStorageInformation.

internalSession string

// daemon vars
lock sync.Mutex
Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Adding missing lock for safety.


// If the retrieval switched to executing, or failed we're done.
if ju.Status != archive.TrackedJobStatusSuccess {
return
Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Weird diff. The main intention is fixing that if this if is true, it will return without calling Commit() below.
The whole thing would work anyway, but it was skipping an intermediate state saving.

Comment on lines +618 to +619
timestamp := time.Unix(le.Res.LogEntry.Time, 0)
ch <- fmt.Sprintf("%s: %s", timestamp.Format("2006-01-02 15:04:05"), le.Res.LogEntry.Message)
Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When giving log messages about the retrieval, include the timestamp as to know how much time has passed between event/messages.

initCmd.Flags().Bool("hard", false, "Discards all local changes if true")
initCmd.Flags().BoolP("yes", "y", false, "Skips the confirmation prompt if true")
initCmd.Flags().BoolP("quiet", "q", false, "Write minimal output")
// (jsign): disabled until this feature is usable in mainnet.
Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All (jsign) reverting, is to re-enable the hidden feature.

services:
powergate:
image: textile/powergate:v2.0.0
image: textile/powergate:v2.3.2-jsign
Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Need to change this to the new docker-image that will be created when textileio/powergate#807 gets merged and tagged.

Comment on lines -298 to -302
// <LOTUS-BUG-HACK>
ccid2, _ := cid.Decode(rootCid1)
err = ipfsPow.Pin().Add(context.Background(), path.IpldPath(ccid2))
// </LOTUS-BUG-HACK>

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The work done in Powergate PR is mostly to workaround this Lotus bug and not depending on a configuration that triggers the bug. So we can delete all this "hacks" that we had in this test.

Comment on lines +23 to +24
func StartPowergate(t util.TestingTWithCleanup) (*pc.Client, *httpapi.HttpApi) {
os.Setenv("LOTUS_ONLINEMODE", strconv.FormatBool(false))
Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We don't use IpfsOnlineMode=true anymore in Lotus for Hub (root problem of all this). So we can simplify here.

Comment on lines +629 to +630
timestamp := time.Unix(le.Res.LogEntry.Time, 0)
ch <- fmt.Sprintf("%s: %s", timestamp.Format("2006-01-02 15:04:05"), le.Res.LogEntry.Message)
Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same as before, also printing dates in hub buck archive watch

@@ -50,6 +50,8 @@ var filQueryMiners = &cobra.Command{
The sort-field flag can take the following values:
Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Basically, add sorting filters for retrievals.

@jsign jsign marked this pull request as ready for review March 22, 2021 19:07
@jsign jsign requested a review from sanderpick March 22, 2021 19:07
Copy link
Copy Markdown
Contributor

@sanderpick sanderpick left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! 🎉

jsign added 5 commits March 22, 2021 16:56
Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>
Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>
Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>
Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>
Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>
Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>
@jsign jsign merged commit 7c51f02 into master Mar 22, 2021
@jsign jsign deleted the jsign/retb branch March 22, 2021 21:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants