Skip to content

Conversation

@calmh
Copy link
Member

@calmh calmh commented Jun 8, 2025

This fixes a problem where remote invalid files (such as files that were once announced but then ignored) would be accounted into the "global size" counters, leading to confusion in the GUI. The handling of such files is slightly different between v1 and v2.

In v1, we let invalid files become the global but avoid adding them to the need list. We then exclude them files from the accounting too, to avoid just the problem I'm solving here.

In v2, we don't have a separate need list but we set a need bit per file, which we already don't do for these files. However we did mark them as "global" which would mean they got accounted. Instead of hacking the accounting, avoid marking them as global as it does not make sense anyway.

Removes a test for an old bug that expects to be able to get an invalid file as the global version. This is no longer and thing (and shouldn't be), and whatever it tested for should ideally not be relevant any more... 😬 That was somehow related to #7474 which is a whole other can of worms (I added a comment there...)

@github-actions github-actions bot added the bug A problem with current functionality, as opposed to missing functionality (enhancement) label Jun 8, 2025
@imsodin
Copy link
Member

imsodin commented Jun 8, 2025

From description and code, it looks like the local file is special in that it is considered global, even if it is invalid - if that's wrong, the following is moot:
It seems inconsistent and afaik also will result in an inflated global count: We don't include invalid files in the local count (at least used to in v1). What's the reason to mark an invalid local file as global, instead of not having a global one like in the case where there's no local?

@calmh
Copy link
Member Author

calmh commented Jun 8, 2025

What's the reason to mark an invalid local file as global, instead of not having a global one like in the case where there's no local?

Good question, and that was also my gut feeling & first fix attempt. We do use the local-is-global-is-invalid scenario in one case, namely receive only folders with local changes. Their behaviour seems to depend on this enough that a couple of tests fail if we don't do it. I didn't want to get too deep into those weeds right now.

It does in any case end up consistent (I think; from what I can see), as we do exclude files with the ignored flag when doing the local counts etc. At least when I test...

@calmh calmh marked this pull request as ready for review June 9, 2025 05:33
@imsodin
Copy link
Member

imsodin commented Jun 10, 2025

Ok, I understand now how this ends up with consistent local and global counts: We do consider local flags in counts, but we don't consider the invalid bool. That's why marking an invalid local file as global is fine, it has local flags, while adding a invalid remote file inflates counts.
Feels a bit icky to go to this half-way state, where some invalid files aren't global, and others/local ones are. We could add a column for invalid to counts to fix the regression/get back the old behaviour until we can mark local invalid as not global too.

And don't we need a migration to remove the global flags with this change from the affected files?

Also long-term/conceptually, I think it's nice-ish to keep marking invalid files as global. That e.g. allows to find all files that are "globally invalid" easily, so we could also drop them from the DB. Given we now clean up old deleted files (still a bit worried about this one :) ), also cleaning up old invalid files seems quite sensible.
And the whole invalid/local-flags split feels a bit inconsistent resp. error prone in handling, I am pondering getting rid of the invalid flag internally and adding FlagLocalRemoteInvalid (converting from that to the invalid flag in the from/to wire functions). Then counting would also work fine, as we already count by local flag.

@calmh
Copy link
Member Author

calmh commented Jun 11, 2025

And don't we need a migration to remove the global flags with this change from the affected files?

... in principle, yes. But we're talking about a few hundred RC users so far. I was going to say reset deltas if this is an actual problem.

I am pondering getting rid of the invalid flag internally and adding FlagLocalRemoteInvalid (converting from that to the invalid flag in the from/to wire functions). Then counting would also work fine, as we already count by local flag.

This sounds like it has merit. Let me look into that (or you can, if you get to it first ;)

@imsodin
Copy link
Member

imsodin commented Jun 11, 2025

This sounds like it has merit. Let me look into that (or you can, if you get to it first ;)

I am already 90% there, started before I saw your message - hope we didn't do it twice now. Still looks like it has merits to me, just a bit a pervasive change :)

@imsodin
Copy link
Member

imsodin commented Jun 11, 2025

Here it is. However take with a good bit of salt, I just pushed this to pass tests without stopping to think overly much along the way, good chance there are some questionable (at best) decisions in there:
#10170

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug A problem with current functionality, as opposed to missing functionality (enhancement)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants