Skip to content

Some writecache fixes#1462

Merged
cthulhu-rider merged 7 commits intonspcc-dev:masterfrom
fyrchik:fix-writecache
Jun 3, 2022
Merged

Some writecache fixes#1462
cthulhu-rider merged 7 commits intonspcc-dev:masterfrom
fyrchik:fix-writecache

Conversation

@fyrchik
Copy link
Contributor

@fyrchik fyrchik commented Jun 1, 2022

  1. Allow to fetch data on demand in FSTree.
  2. Initialize flush mark LRU cache on startup.

@codecov
Copy link

codecov bot commented Jun 1, 2022

Codecov Report

Merging #1462 (e58c483) into master (021aa97) will decrease coverage by 0.03%.
The diff coverage is 43.47%.

❗ Current head e58c483 differs from pull request most recent head b526394. Consider uploading reports for the commit b526394 to get more accurate results

@@            Coverage Diff             @@
##           master    #1462      +/-   ##
==========================================
- Coverage   36.36%   36.32%   -0.04%     
==========================================
  Files         305      305              
  Lines       17782    17788       +6     
==========================================
- Hits         6466     6462       -4     
- Misses      10774    10778       +4     
- Partials      542      548       +6     
Impacted Files Coverage Δ
pkg/local_object_storage/blobstor/blobovnicza.go 66.30% <16.66%> (-0.36%) ⬇️
pkg/local_object_storage/blobovnicza/control.go 73.91% <42.85%> (-7.49%) ⬇️
pkg/local_object_storage/blobstor/fstree/fstree.go 69.72% <53.33%> (-3.28%) ⬇️
...kg/local_object_storage/blobovnicza/blobovnicza.go 82.85% <100.00%> (-0.48%) ⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 021aa97...b526394. Read the comment docs.

carpawell
carpawell previously approved these changes Jun 2, 2022
fyrchik added 7 commits June 3, 2022 13:58
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
If we should process address based on some condition, there is no need
to read file content in memory.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
Some of the objects are already flushed, don't do it twice.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
`Update` becomes a botleneck under a heavy load.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
`log.With` is suitable during initialization, but in other places it induces
some overhead, even when branches with logging are not taken.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
@cthulhu-rider cthulhu-rider merged commit feef9a9 into nspcc-dev:master Jun 3, 2022
cthulhu-rider pushed a commit that referenced this pull request Jun 3, 2022
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
cthulhu-rider pushed a commit that referenced this pull request Jun 3, 2022
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
cthulhu-rider pushed a commit that referenced this pull request Jun 3, 2022
If we should process address based on some condition, there is no need
to read file content in memory.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
cthulhu-rider pushed a commit that referenced this pull request Jun 3, 2022
Some of the objects are already flushed, don't do it twice.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
cthulhu-rider pushed a commit that referenced this pull request Jun 3, 2022
`Update` becomes a botleneck under a heavy load.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
cthulhu-rider pushed a commit that referenced this pull request Jun 3, 2022
`log.With` is suitable during initialization, but in other places it induces
some overhead, even when branches with logging are not taken.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
aprasolova pushed a commit to aprasolova/neofs-node that referenced this pull request Oct 19, 2022
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
aprasolova pushed a commit to aprasolova/neofs-node that referenced this pull request Oct 19, 2022
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
aprasolova pushed a commit to aprasolova/neofs-node that referenced this pull request Oct 19, 2022
If we should process address based on some condition, there is no need
to read file content in memory.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
aprasolova pushed a commit to aprasolova/neofs-node that referenced this pull request Oct 19, 2022
Some of the objects are already flushed, don't do it twice.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
aprasolova pushed a commit to aprasolova/neofs-node that referenced this pull request Oct 19, 2022
`Update` becomes a botleneck under a heavy load.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
aprasolova pushed a commit to aprasolova/neofs-node that referenced this pull request Oct 19, 2022
`log.With` is suitable during initialization, but in other places it induces
some overhead, even when branches with logging are not taken.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
aprasolova pushed a commit to aprasolova/neofs-node that referenced this pull request Oct 19, 2022
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
roman-khimov added a commit that referenced this pull request Jun 24, 2025
The problem is that it can take a significant amount of time to perform while
the value is questionable.

It was initially added in 8717820 (#1462)
when write cache had completely different structure and function.
733a792 tried to optimize it in #2068, but
most of the code was subsequently deleted and as write cache is no longer a
read cache this lead to the remaining code doing only the following:
 * checking objects against the metabase
 * checking objects against the underlying long-term storage

If the first one is ever triggered (we detect non-existent object) this just
means WC is out of sync with metabase which can happen for a number of other
reasons. Lower level blobstor can also get out of sync with metabase in some
cases, so it's not like this is something unique. If we're to drop this check
we'd just flush some dead object, but then GC should still be able to remove
it if it's deleted normally. If it's some other type of race --- like WC
accepted an object and metabase not yet then we shouldn't delete it at all
for safety reasons.

Then if some valid object was previously flushed already we'll retry flushing
it again and it's not a problem, EEXIST is not an error (but #2563), so we'd
waste some time, but it's not very likely we're going to since this case is
rare.

So it should be safe to drop this useless code.

Signed-off-by: Roman Khimov <roman@nspcc.ru>
roman-khimov added a commit that referenced this pull request Jun 24, 2025
The problem is that it can take a significant amount of time to perform while
the value is questionable.

It was initially added in 8717820 (#1462)
when write cache had completely different structure and function.
733a792 tried to optimize it in #2068, but
most of the code was subsequently deleted and as write cache is no longer a
read cache this lead to the remaining code doing only the following:
 * checking objects against the metabase
 * checking objects against the underlying long-term storage

If the first one is ever triggered (we detect non-existent object) this just
means WC is out of sync with metabase which can happen for a number of other
reasons. Lower level blobstor can also get out of sync with metabase in some
cases, so it's not like this is something unique. If we're to drop this check
we'd just flush some dead object, but then GC should still be able to remove
it if it's deleted normally. If it's some other type of race --- like WC
accepted an object and metabase not yet then we shouldn't delete it at all
for safety reasons.

Then if some valid object was previously flushed already we'll retry flushing
it again and it's not a problem, EEXIST is not an error (but #2563), so we'd
waste some time, but it's not very likely we're going to since this case is
rare.

So it should be safe to drop this useless code.

Signed-off-by: Roman Khimov <roman@nspcc.ru>
roman-khimov added a commit that referenced this pull request Jun 24, 2025
The problem is that it can take a significant amount of time to perform while
the value is questionable.

It was initially added in 8717820 (#1462)
when write cache had completely different structure and function.
733a792 tried to optimize it in #2068, but
most of the code was subsequently deleted and as write cache is no longer a
read cache this lead to the remaining code doing only the following:
 * checking objects against the metabase
 * checking objects against the underlying long-term storage

If the first one is ever triggered (we detect non-existent object) this just
means WC is out of sync with metabase which can happen for a number of other
reasons. Lower level blobstor can also get out of sync with metabase in some
cases, so it's not like this is something unique. If we're to drop this check
we'd just flush some dead object, but then GC should still be able to remove
it if it's deleted normally. If it's some other type of race --- like WC
accepted an object and metabase not yet then we shouldn't delete it at all
for safety reasons.

Then if some valid object was previously flushed already we'll retry flushing
it again and it's not a problem, EEXIST is not an error (but #2563), so we'd
waste some time, but it's not very likely we're going to since this case is
rare.

So it should be safe to drop this useless code.

Signed-off-by: Roman Khimov <roman@nspcc.ru>
roman-khimov added a commit that referenced this pull request Jun 24, 2025
The problem is that it can take a significant amount of time to perform while
the value is questionable.

It was initially added in 8717820 (#1462)
when write cache had completely different structure and function.
733a792 tried to optimize it in #2068, but
most of the code was subsequently deleted and as write cache is no longer a
read cache this lead to the remaining code doing only the following:
 * checking objects against the metabase
 * checking objects against the underlying long-term storage

If the first one is ever triggered (we detect non-existent object) this just
means WC is out of sync with metabase which can happen for a number of other
reasons. Lower level blobstor can also get out of sync with metabase in some
cases, so it's not like this is something unique. If we're to drop this check
we'd just flush some dead object, but then GC should still be able to remove
it if it's deleted normally. If it's some other type of race --- like WC
accepted an object and metabase not yet then we shouldn't delete it at all
for safety reasons.

Then if some valid object was previously flushed already we'll retry flushing
it again and it's not a problem, EEXIST is not an error (but #2563), so we'd
waste some time, but it's not very likely we're going to since this case is
rare.

So it should be safe to drop this useless code.

Signed-off-by: Roman Khimov <roman@nspcc.ru>
roman-khimov added a commit that referenced this pull request Jun 25, 2025
The problem is that it can take a significant amount of time to perform while
the value is questionable.

It was initially added in 8717820 (#1462)
when write cache had completely different structure and function.
733a792 tried to optimize it in #2068, but
most of the code was subsequently deleted and as write cache is no longer a
read cache this lead to the remaining code doing only the following:
 * checking objects against the metabase
 * checking objects against the underlying long-term storage

If the first one is ever triggered (we detect non-existent object) this just
means WC is out of sync with metabase which can happen for a number of other
reasons. Lower level blobstor can also get out of sync with metabase in some
cases, so it's not like this is something unique. If we're to drop this check
we'd just flush some dead object, but then GC should still be able to remove
it if it's deleted normally. If it's some other type of race --- like WC
accepted an object and metabase not yet then we shouldn't delete it at all
for safety reasons.

Then if some valid object was previously flushed already we'll retry flushing
it again and it's not a problem, EEXIST is not an error (but #2563), so we'd
waste some time, but it's not very likely we're going to since this case is
rare.

So it should be safe to drop this useless code.

Signed-off-by: Roman Khimov <roman@nspcc.ru>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants