Skip to content

descs,catkv: rewrite Collection backend#87067

Merged
craig[bot] merged 6 commits intocockroachdb:masterfrom
postamar:spin-off-stored-descriptors
Aug 31, 2022
Merged

descs,catkv: rewrite Collection backend#87067
craig[bot] merged 6 commits intocockroachdb:masterfrom
postamar:spin-off-stored-descriptors

Conversation

@postamar
Copy link
Copy Markdown

This rewrite was motivated by the upcoming need to validate descriptors
up to different levels depending on why they were being read. Doing so
requires tracking the validation level in a more granular fashion within
the Collection's caches. Trying to make this work within the existing
scheme where the uncommitted and stored descriptors are all in the same
layer turned out to be painful and hard to reason about.

This commit does several things:

  1. The uncommitted descriptors are spun off into their own distinct layer
    again.
  2. The stored descriptors now mirror what's in storage and so that whole
    layer could effectively be moved to catkv.
  3. Execution flow through the layers in the Collection is now
    straightforwardly implemented and easy to reason about. Validation and
    hydration now take place in unique, well-defined steps at the very end
    of a lookup, be it by name or by ID.
  4. Instances of this new stored descriptors layer are now also used by
    the lease manager, the Direct() interface provided by the Collection,
    etc.
  5. As a result the logic in catkv could be significantly cleaned up.
    In particular, lookups for descriptor validation are significantly
    less convoluted now.
  6. The SystemNamespaceCache object was also moved to catkv and now
    supports caching descriptors, and so was renamed to
    SystemCatalogCache.

Despite significant rewriting of the inner workings of the descriptors
collection, this commit should not functionally change anything.

Informs #85263.

Release justification: low-risk, high-benefit change.
Release note: None

@cockroach-teamcity
Copy link
Copy Markdown
Member

This change is Reviewable

@postamar postamar force-pushed the spin-off-stored-descriptors branch 5 times, most recently from 8dd5006 to eebff4d Compare August 30, 2022 01:58
@postamar postamar marked this pull request as ready for review August 30, 2022 02:49
@postamar postamar requested review from a team and rhu713 August 30, 2022 02:49
@postamar postamar force-pushed the spin-off-stored-descriptors branch from eebff4d to baefec5 Compare August 30, 2022 02:51
Copy link
Copy Markdown
Contributor

@ajwerner ajwerner left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is nice. The only major hesitation I have is around the SystemDatabaseCache. The rest of my comments are minor.

Reviewed 1 of 1 files at r1, 34 of 34 files at r2, all commit messages.
Reviewable status: :shipit: complete! 0 of 0 LGTMs obtained (waiting on @postamar and @rhu713)


pkg/sql/catalog/descs/descriptor.go line 225 at r2 (raw file):

	//
	// TODO(ajwerner): More generally leverage this set of kv descriptors on
	// the resolution path.

I think I'd say Jason achieved this TODO.

Code quote:

	// TODO(ajwerner): More generally leverage this set of kv descriptors on
	// the resolution path.

pkg/sql/catalog/descs/descriptor.go line 338 at r2 (raw file):

// and (2) if mutable descriptors are requested, these are present in the
// uncommitted descriptors layer.
// Known validation levels can be provided via validationLevels, otherwise when

nit: my brain had a hard time parsing otherwise when nil safe defaults are used instead. Can you sprinkle in a comma or two?

Code quote:

// Known validation levels can be provided via validationLevels, otherwise when
// nil safe defaults are used instead. In any case, after validation is

pkg/sql/catalog/descs/descriptor.go line 348 at r2 (raw file):

	validationLevels []catalog.ValidationLevel,
) error {
	if validationLevels == nil {

add a defensive check that if validationLevels is not nil that it has the same length as descs?


pkg/sql/catalog/descs/uncommitted_descriptors.go line 181 at r2 (raw file):

		return errors.Wrap(err, "Memory usage exceeds limit for uncommittedDescriptors")
	}
	ud.mutable.Upsert(mut, true /* skipNameMap */)

why are we unconditionally skipping the name map? It feels like if we're going to always skip the nameMap we should use a data structure that does not have one.


pkg/sql/catalog/internal/catkv/stored_catalog.go line 38 at r2 (raw file):

	// cache mirrors the descriptors in storage.
	// This map does not store descriptors by name.

I said it elsewhere: let's export an object from nstree which wraps the existing byIDMap and avoid the confusion this thing is causing


pkg/sql/catalog/internal/catkv/system_database_cache.go line 34 at r2 (raw file):

// the lease manager to cache all system table IDs.
type SystemDatabaseCache struct {
	syncutil.RWMutex

I don't think it's right to export the mutex methods. The style here is generally to embed the mutex inside of an anonymous, embedded struct. I think that would work nicely here.

type SystemDatabaseCache struct {
	mu struct {
		syncutil.RWMutex
		m map[roachpb.Version]*nstree.MutableCatalog
        }
}

pkg/sql/catalog/internal/catkv/system_database_cache.go line 35 at r2 (raw file):

type SystemDatabaseCache struct {
	syncutil.RWMutex
	m map[roachpb.Version]*nstree.MutableCatalog

The semantics of the versions in use here are interesting. I think that right now, it doesn't really matter because we never rename something in the system database and we only use this for caching names, right?


pkg/sql/catalog/internal/catkv/system_database_cache.go line 107 at r2 (raw file):

// Anything not pertaining to the system database is ignored.
// Presently, system descriptors are all ignored and only namespace entries are
// used.

what's your vision here? My plan for 23.1 is to start treating system descriptors, to the extent possible, like regular descriptors and not to make them more special. We'll need this to allow for deeper configuration of multi-region primitives

Code quote:

// Presently, system descriptors are all ignored and only namespace entries are
// used.

Recent changes to the descriptors collection have uncovered some
brittleness in the logic for writing namespace entries for dropped
descriptors when creating backups.

This commit makes it:
- write deletion entries for descriptors other than tables, preventing
  possible catalog corruption.
- throw a better error message when attempting to write two identical
  keys consecutively.

Release justification: low-risk change
Release note: None
Copy link
Copy Markdown
Author

@postamar postamar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the review.

Reviewable status: :shipit: complete! 0 of 0 LGTMs obtained (waiting on @ajwerner, @postamar, and @rhu713)


pkg/sql/catalog/descs/descriptor.go line 225 at r2 (raw file):

Previously, ajwerner wrote…

I think I'd say Jason achieved this TODO.

Removing.


pkg/sql/catalog/descs/descriptor.go line 338 at r2 (raw file):

Previously, ajwerner wrote…

nit: my brain had a hard time parsing otherwise when nil safe defaults are used instead. Can you sprinkle in a comma or two?

Rewriting.


pkg/sql/catalog/descs/descriptor.go line 348 at r2 (raw file):

Previously, ajwerner wrote…

add a defensive check that if validationLevels is not nil that it has the same length as descs?

Will do.


pkg/sql/catalog/descs/uncommitted_descriptors.go line 181 at r2 (raw file):

Previously, ajwerner wrote…

why are we unconditionally skipping the name map? It feels like if we're going to always skip the nameMap we should use a data structure that does not have one.

Will do.


pkg/sql/catalog/internal/catkv/stored_catalog.go line 38 at r2 (raw file):

Previously, ajwerner wrote…

I said it elsewhere: let's export an object from nstree which wraps the existing byIDMap and avoid the confusion this thing is causing

Will do. Agreed.


pkg/sql/catalog/internal/catkv/system_database_cache.go line 34 at r2 (raw file):

Previously, ajwerner wrote…

I don't think it's right to export the mutex methods. The style here is generally to embed the mutex inside of an anonymous, embedded struct. I think that would work nicely here.

type SystemDatabaseCache struct {
	mu struct {
		syncutil.RWMutex
		m map[roachpb.Version]*nstree.MutableCatalog
        }
}

Whoops, right. Will do.


pkg/sql/catalog/internal/catkv/system_database_cache.go line 35 at r2 (raw file):

Previously, ajwerner wrote…

The semantics of the versions in use here are interesting. I think that right now, it doesn't really matter because we never rename something in the system database and we only use this for caching names, right?

Indeed it's just me being defensive. Adding commentary.


pkg/sql/catalog/internal/catkv/system_database_cache.go line 107 at r2 (raw file):

Previously, ajwerner wrote…

what's your vision here? My plan for 23.1 is to start treating system descriptors, to the extent possible, like regular descriptors and not to make them more special. We'll need this to allow for deeper configuration of multi-region primitives

My vision is the same as yours. Adding commentary to explain why we don't actually cache system descriptors other than the system DB.

@postamar postamar force-pushed the spin-off-stored-descriptors branch 2 times, most recently from 3119e59 to 803ab34 Compare August 31, 2022 15:39
Marius Posta added 2 commits August 31, 2022 12:22
This rewrite was motivated by the upcoming need to validate descriptors
up to different levels depending on why they were being read. Doing so
requires tracking the validation level in a more granular fashion within
the Collection's caches. Trying to make this work within the existing
scheme where the uncommitted and stored descriptors are all in the same
layer turned out to be painful and hard to reason about.

This commit does several things:
1. The uncommitted descriptors are spun off into their own distinct layer
   again.
2. The stored descriptors now mirror what's in storage and so that whole
   layer could effectively be moved to catkv.
3. Execution flow through the layers in the Collection is now
   straightforwardly implemented and easy to reason about. Validation and
   hydration now take place in unique, well-defined steps at the very end
   of a lookup, be it by name or by ID.
3. Instances of this new stored descriptors layer are now also used by
   the lease manager, the Direct() interface provided by the Collection,
   etc.
4. As a result the logic in catkv could be significantly cleaned up.
   In particular, lookups for descriptor validation are significantly
   less convoluted now.
5. The SystemNamespaceCache object was also moved to catkv, renamed to
   SystemDatabaseCache and now caches based on the cluster version.

Despite significant rewriting of the inner workings of the descriptors
collection, this commit should not functionally change anything.

Informs cockroachdb#85263.

Release justification: low-risk, high-benefit change.
Release note: None
This commit adds a new nstree.Map-like data structure without any
by-name operations.

Release justification: low-risk change which does not change
functionality
Release note: None
Copy link
Copy Markdown
Contributor

@ajwerner ajwerner left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:lgtm: my comments are nits, but in general, I'd prefer we not export methods unless we need them to conform to some interface explicitly. I spent a little while looking around for such an interface.

The internal executor change I sort of get, but I don't fully get. I feel like we can tighten up the code and commentary around that change.

Reviewed 3 of 34 files at r3, 19 of 34 files at r4, 8 of 17 files at r5, 1 of 2 files at r6, 9 of 11 files at r7, all commit messages.
Reviewable status: :shipit: complete! 1 of 0 LGTMs obtained (waiting on @ajwerner, @postamar, and @rhu713)


-- commits line 69 at r6:

which it never has when using the planner's state.

Is that true? Consider:

return ie.WithSyntheticDescriptors(

return validateUniqueWithoutIndexConstraintInTxn(

if err := runSchemaChangesInTxn(

Code quote:

  prevent that by attaching the logic to reset the synthetic descriptors to
  whether the internal executor has attached synthetic descriptors, which
  it never has when using the planner's state.

-- commits line 5 at r7:
I anticipate this all worked because the backups don't store dropped descriptors. Does this commit fix any known bug? Should it be in this PR or should it be pulled out into a separate one such that it might be tested?

Code quote:

  Recent changes to the descriptors collection have uncovered some
  brittleness in the logic for writing namespace entries for dropped
  descriptors when creating backups.

pkg/sql/internal.go line 337 at r6 (raw file):

			ex.extraTxnState.descCollection = ie.extraTxnState.descCollection
			ex.extraTxnState.fromOuterTxn = true
			ex.extraTxnState.shouldResetSyntheticDescriptors = len(ie.syntheticDescriptors) > 0

can you add more commentary here?


pkg/sql/internal.go line 334 at r7 (raw file):

	// and job collections from the caller.
	postSetupFn := func(ex *connExecutor) {
		if ie.extraTxnState != nil && ie.extraTxnState.descCollection != nil {

what's the case where extraTxnState is not nil but descCollection is? Also, while you're here, if you have to do another revision, rename to descsCollection?


pkg/sql/catalog/descs/synthetic_descriptors.go line 23 at r7 (raw file):

}

func (sd *syntheticDescriptors) Add(desc catalog.Descriptor) {

why are all these methods now exported?


pkg/sql/catalog/descs/uncommitted_descriptors.go line 63 at r7 (raw file):

// Reset zeroes the object for re-use in a new transaction.
func (ud *uncommittedDescriptors) Reset(ctx context.Context) {

Why are all of the methods on this unexported struct exported? It doesn't seem like it's embedded anywhere

@postamar postamar force-pushed the spin-off-stored-descriptors branch from 0686a75 to c3af318 Compare August 31, 2022 20:49
…otrs

The pre-existing logic would reset the synthetic descriptors whenever anything
was run on behalf of the internal planner. That's not what we want. We
prevent that by attaching the logic to reset the synthetic descriptors to
whether the internal executor has attached synthetic descriptors, which
it never has when using the declarative schema changer.

Release justification: part of bug fix

Release note: None
This allows synthetic descriptors to show up in virtual tables.

Release justification: part of bug fix.

Release note: None
This commit doesn't otherwise change anything.

Release justification: very low-risk change
Release note: None
@postamar postamar force-pushed the spin-off-stored-descriptors branch from c3af318 to 286d828 Compare August 31, 2022 21:30
@postamar
Copy link
Copy Markdown
Author

Thanks for the review! I unexported the methods and other such things.

bors r+

@craig
Copy link
Copy Markdown
Contributor

craig bot commented Aug 31, 2022

Build succeeded:

@craig craig bot merged commit b272795 into cockroachdb:master Aug 31, 2022
@postamar postamar deleted the spin-off-stored-descriptors branch September 1, 2022 00:21
craig bot pushed a commit that referenced this pull request Sep 8, 2022
86420: catalog: spin off and adopt back-reference validation level r=postamar a=postamar

Previously, forward- and backward-reference-checking was done in one
same validation phase, identified by the
catalog.ValidationLevelCrossReferences constant and a similarly named
method on catalog.Descriptor. This validation phase was exercised both
when reading and when writing descriptors.

The problem with this was that a corrupt back-reference would make
a table unusable until descriptor surgery was performed. While this
might be warranted in the context of schema changes, that's not the case
when it comes to queries, which don't really care about these
back-references. For this reason, we want to bypass back-reference
validation when reading descriptors for the purpose of serving queries.
These reads are characterized by the flag AvoidLeased being unset.

To do this, this commit splits the cross-reference validation level into
two. The back-reference level now exists to check the integrity of
back-references directly, as well as to check that forward references
also have matching back-references in the referenced descriptors.

Fixes #85263.

Release justification: low-risk, high benefit change
Release note: None

---- 

This PR is based on #87067 which is the product of what made sense for me to take action on right now regarding #86420 (comment) .



Co-authored-by: Marius Posta <marius@cockroachlabs.com>
postamar pushed a commit to postamar/cockroach that referenced this pull request Sep 9, 2022
Recent work on the descs.Collection (PR cockroachdb#87067) introduced a regression
in which it would return bad lease.IDVersion versions for the uncommitted
descriptors.

Fixes cockroachdb#87672.

Release justification: important bug fix
Release note: None
postamar pushed a commit to postamar/cockroach that referenced this pull request Sep 9, 2022
Recent work on the descs.Collection (PR cockroachdb#87067) introduced a regression
in which it would return bad lease.IDVersion versions for the uncommitted
descriptors.

In addition to fixing this, this commit corrects the lifecycle of the
`original` descriptors in the uncommitted layer, which now remain
unaffected by changes in the storage layer.

This commit tightens the constraints on what can be passed to
AddUncommittedDescriptor.

Fixes cockroachdb#87672.

Release justification: important bug fix
Release note: None
craig bot pushed a commit that referenced this pull request Sep 9, 2022
87525: sql: fix recording db names with hyphen for idx usage stats r=xinhaoz a=xinhaoz

Fixes #85577

This commit fixes a bug where the query constructed during
index usage stats recording failed for db names containing
a hyphen. The db name placeholder is now converted to string
from the `tree.Name`, which should properly escape hyphen
characters for the query.

Release justification: bug fix

Release note (bug fix): index usage stats are properly captured for database names with hyphens

87706: descs: fix txn commit waiting on wrong lease version r=postamar a=postamar

Recent work on the descs.Collection (PR #87067) introduced a regression
in which it would return bad lease.IDVersion versions for the uncommitted
descriptors.

Fixes #87672.

Release justification: important bug fix
Release note: None

Co-authored-by: Xin Hao Zhang <xzhang@cockroachlabs.com>
Co-authored-by: Marius Posta <marius@cockroachlabs.com>
blathers-crl bot pushed a commit that referenced this pull request Sep 9, 2022
Recent work on the descs.Collection (PR #87067) introduced a regression
in which it would return bad lease.IDVersion versions for the uncommitted
descriptors.

In addition to fixing this, this commit corrects the lifecycle of the
`original` descriptors in the uncommitted layer, which now remain
unaffected by changes in the storage layer.

This commit tightens the constraints on what can be passed to
AddUncommittedDescriptor.

Fixes #87672.

Release justification: important bug fix
Release note: None
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants