Skip to content

fix: replace pre-existing pins for MFS root#7864

Merged
aschmahmann merged 9 commits intopetar/pinmfsfrom
fix/replace-duplicate-mfs-pins
Jan 21, 2021
Merged

fix: replace pre-existing pins for MFS root#7864
aschmahmann merged 9 commits intopetar/pinmfsfrom
fix/replace-duplicate-mfs-pins

Conversation

@lidel
Copy link
Member

@lidel lidel commented Jan 15, 2021

This is a refactor of #7798 that replaces the existing MFS root pin (if present) instead of creating a new pin.

  • find by name all pre-existing pins across all Statuses (not just 'pinned')
  • replace pre-existing pin (or create a new one if none found)

Without this we would be adding duplicate pins when pinning service
is still processing previous pin request (queued, pinning):

QmV6QnQNuyJFq88fSVVY9u5fxftQEKu6unsD62QpLQSefP  queued  policy/12D3KooWEcsoib3zFds4USH9bASPWBvJHRvfHYbxjR8QCHQ7y1vj/mfs
QmbnBveHJSqcqmVK8UE64riotVzMGirCKYE1SQLXcG25dT  queued  policy/12D3KooWEcsoib3zFds4USH9bASPWBvJHRvfHYbxjR8QCHQ7y1vj/mfs
Qmf1FHNVp74F4TGtNg1myXZgaV4ApwmYgC8iXcUX8cidDy  queued  policy/12D3KooWEcsoib3zFds4USH9bASPWBvJHRvfHYbxjR8QCHQ7y1vj/mfs
QmXnVAeJ9braE7ho7hdPqkmd9cB2ewKtzXNcFFFm3W32EA  queued  policy/12D3KooWEcsoib3zFds4USH9bASPWBvJHRvfHYbxjR8QCHQ7y1vj/mfs
QmeSUtG78SnqoCLDnZw5kYx2w9QnCp8JhkKHxK537y2JET  queued  policy/12D3KooWEcsoib3zFds4USH9bASPWBvJHRvfHYbxjR8QCHQ7y1vj/mfs
Qmacb2NPrz1tzpB3GarzqaYuLb54o71iJAAL61jGbmJEmu  queued  policy/12D3KooWEcsoib3zFds4USH9bASPWBvJHRvfHYbxjR8QCHQ7y1vj/mfs
QmRk9WAfArEpEPw6xc5hTUmKPeRM85fUCJoSqGPWYfMCB4  queued  policy/12D3KooWEcsoib3zFds4USH9bASPWBvJHRvfHYbxjR8QCHQ7y1vj/mfs
QmVYD7stnRgG4HHgrnkpbCUvLTvjLmZbhTJNe5ZNx7bDmh  queued  policy/12D3KooWEcsoib3zFds4USH9bASPWBvJHRvfHYbxjR8QCHQ7y1vj/mfs
QmTiCXsj9nwFCDpUxq3fAUvZW49s3FCHPtxb8KeYoGfmri  queued  policy/12D3KooWEcsoib3zFds4USH9bASPWBvJHRvfHYbxjR8QCHQ7y1vj/mfs

Notes

  • Replace at Pinata will still produce duplicate pins, but for a different reason:
    • This is temporary bug on their end and should not block us from switching to Replace method.
    • Our CI should be green, because we don't use Pinata for tests there, and Replace from rb-pinning-service-api works as expected.

@lidel lidel requested review from aschmahmann and petar January 15, 2021 12:29
existingRequestID := "" // is there any pre-existing MFS pin with pinName (for any CID)?
alreadyPinned := false // is CID for current MFS already pinned?
for ps := range lsPinCh {
existingRequestID = ps.GetRequestId()
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IIUC this means we're replacing the oldest pin with the correct name. This doesn't seem unreasonable, but it's non-obvious so I wanted to clarify that this is the intended behavior.

Copy link
Member Author

@lidel lidel Jan 15, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correct. The assumption is that the name is unique enough to not have more than a single pre-existing one, but if for some reason there is more, we just replace the last one we inspected.

lidel added a commit that referenced this pull request Jan 16, 2021
lidel added 9 commits January 16, 2021 01:41
This illustrates that logic responsible for finding existing MFS root pin
does not account for pins in states other than "pinned".
- find pre-existing pin across all Statuses (not just 'pinned')
- replace pre-existing pin or create a new one

Without this we would be adding duplicate pins when pinning service
is still processing previos pin request (queued, pinning).
If MFS root pin already exists for the CID but has Failed state we Replace it
with the same CID to trigger a retry.
iiuc node was sending unnecessary HTTP requests to remote service,
this makes sure we skip MFS pin logic if CID did not change
This ensures we use PinStatus.created during the RepinInterval check
Pinning Service may operate in different timezone. Spec suggests use of
UTC everywhere, but in case we got different timezone, we normalize to
UTC for easier debugging.
This confirms pin gets replaced with new CID when MFS root changes
@lidel lidel force-pushed the fix/replace-duplicate-mfs-pins branch from a26a252 to 1f3f68b Compare January 16, 2021 00:46
@lidel
Copy link
Member Author

lidel commented Jan 16, 2021

Quick update:

  • When pin already exists, we now use creation time returned by remote service instead of time.Now()
  • Added sharness test that confirms the MFS root pin gets replaced with new CID when MFS is modified.
  • Normalized timestamps to UTC for easier debugging across services
    (Pinning Services API defaults to UTC, but this makes it explicit)
  • Tweaked conditions and logged strings a bit to produce human-readable logs
  • (keeping this as a draft until I am able to test Replace operation against Pinata)

@lidel lidel marked this pull request as ready for review January 20, 2021 23:29
@lidel
Copy link
Member Author

lidel commented Jan 20, 2021

@petar @aschmahmann Pinata will fix remaining issue on their end (not a blocker for us).
I propose we merge these fixes/changes into #7798 so we avoid split-brain reviews and tests going forward.

@aschmahmann aschmahmann merged commit 58898cc into petar/pinmfs Jan 21, 2021
@aschmahmann
Copy link
Contributor

merging as @petar gave the 👍. We can continue on the other PR.

@lidel lidel deleted the fix/replace-duplicate-mfs-pins branch January 21, 2021 23:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants