Skip to content

config: introduce pseudo "tenants" zone#49784

Merged
craig[bot] merged 5 commits intocockroachdb:masterfrom
nvb:nvanbenschoten/tenantSplit
Jun 4, 2020
Merged

config: introduce pseudo "tenants" zone#49784
craig[bot] merged 5 commits intocockroachdb:masterfrom
nvb:nvanbenschoten/tenantSplit

Conversation

@nvb
Copy link
Copy Markdown
Contributor

@nvb nvb commented Jun 2, 2020

Fixes #49318.
Fixes #49445.
Progress towards #48123.
Informs #48774.

This commit introduces a new pseudo object ID in the system tenant's namespace called "tenants". Like "liveness" and "timeseries" before it, the pseudo object allows zone configurations to be applied to pseudo-objects that do not live directly in the system tenant's SQL keyspace. In this case, the "tenants" zone allows zone configurations to be set by the system tenant and applied to all other tenants in the system. There may come a time when we want secondary tenants to have more control over their zone configurations, but that will require a much larger change to the zone config structure and UX as a whole.

While making this change, we rationalize the rest of zone configuration handling and how it relates to multi-tenancy. Now that secondary tenant ranges have a zone config to call their own, we can make sense of calls from SQL into the zone configuration infrastructure. We gate off calls that don't make sense for secondary tenants and clean up hooks in SQL that handle zone config manipulation.

All of this works towards a good cause - we eliminate the remaining uses of keys.TODOSQLCodec from pkg/sql/... and pkg/config/..., bringing us a big step closer towards being able to remove the placeholder and close #48123.

This work also reveals that in order to address #48774, we need to be able to determine split points from the SystemConfig. This makes it very difficult to split on secondary tenant object (e.g. table) boundaries. However, it makes it straightforward to split on secondary tenant keysapce boundaries. This is already what we were thinking (see #47907), so out both convenience and desire, I expect that we'll follow this up with a PR that splits Ranges only at secondary tenant boundaries - placing the overhead of an otherwise empty tenant at only a single Range and a few KBs of data.

@nvb nvb requested review from a team, asubiotto and tbg June 2, 2020 03:56
@nvb nvb requested a review from a team as a code owner June 2, 2020 03:56
@nvb nvb requested review from dt and removed request for a team June 2, 2020 03:56
@cockroach-teamcity
Copy link
Copy Markdown
Member

This change is Reviewable

@nvb nvb force-pushed the nvanbenschoten/tenantSplit branch from 00de2c4 to 29a399d Compare June 2, 2020 04:32
Copy link
Copy Markdown
Contributor

@asubiotto asubiotto left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Glad to see this! Do we have an issue to track post phase 2 work for tenant zone configs once #49445 is closed? We should create one if not.

I'll rerun the logic tests to see which blacklists can be removed.

Reviewed 1 of 1 files at r1, 4 of 4 files at r2, 5 of 5 files at r3, 44 of 44 files at r4.
Reviewable status: :shipit: complete! 0 of 0 LGTMs obtained (waiting on @dt, @nvanbenschoten, and @tbg)


pkg/sql/drop_table.go, line 354 at r4 (raw file):

	// this faster mechanism.
	if tableDesc.IsTable() && !tableDesc.IsInterleaved() {
		// TODO BEFORE MERGING: confirm that we can delete this.

I think so. Based on 7abac2f, this code was added as a sanity check. OOC, how would this inability to get a zone config for a dropped tenant table have manifested itself otherwise? Would the default TTL be applied?


pkg/sql/logictest/testdata/logic_test/tenant_unsupported, line 37 at r4 (raw file):

# https://github.com/cockroachdb/cockroach/issues/49318 is fixed.
# statement error operation is unsupported
# ALTER TABLE kv CONFIGURE ZONE USING num_replicas = 123

You can probably delete this.

Copy link
Copy Markdown
Member

@tbg tbg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:lgtm_strong: looks good!

Now that secondary tenant ranges have a zone config to call their own, we can make sense of calls from SQL into the zone configuration infrastructure. We gate off calls that don't make sense for secondary tenants and clean up hooks in SQL that handle zone config manipulation.

This left me slightly confused. Secondary tenants don't really get to do anything with the zone configs, or am I missing something? All they will see is the default zone config (configured at startup). Your description made it sound as if tenants can do... something with the zone config, but I don't think you even let them list the "background" config that applies to them.

Reviewed 1 of 1 files at r1, 4 of 4 files at r2, 5 of 5 files at r3, 44 of 44 files at r4.
Reviewable status: :shipit: complete! 1 of 0 LGTMs obtained (waiting on @dt and @nvanbenschoten)


pkg/config/system.go, line 207 at r3 (raw file):

	// No maximum specified; maximum ID is the last entry in the descriptor
	// table.

or the largest pseudo ID, whichever is larger.


pkg/config/system.go, line 221 at r3 (raw file):

	// Maximum specified: need to search the descriptor table. Binary search
	// through all descriptor table values to find the first descriptor with ID
	// >= maxID.

and pick either it or maxPseudoID.


pkg/config/system.go, line 229 at r3 (raw file):

		}
		var id uint32
		id, err = keys.TODOSQLCodec.DecodeDescMetadataID(searchSlice[i].Key)

This can't be intentional, at least not without a comment.


pkg/keys/constants.go, line 399 at r4 (raw file):

// PseudoTableIDs is the list of ids from above that are not real tables (i.e.
// there's no table descriptor). They're grouped here because the cluster
// bootstrap process needs to create splits for them; splits for the tables

Can you revisit the "because the bootstrap process needs to create splits for them"? I think I remember that it does create splits for the tables with these IDs (which don't exist). I think something is fishy here, it's not like this package has any notion of where the split points that correspond to these pseudoIDs are. Probably the comment makes no sense.


pkg/sql/zone_config.go, line 159 at r2 (raw file):

func ZoneConfigHook(
	cfg *config.SystemConfig, id uint32,
) (*zonepb.ZoneConfig, *zonepb.ZoneConfig, bool, error) {

While you're here, explain the retvals?

Copy link
Copy Markdown
Contributor Author

@nvb nvb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TFTRs!

Do we have an issue to track post phase 2 work for tenant zone configs once #49445 is closed?

I opened #49854 to track this.

This left me slightly confused.

Reworded.

Reviewable status: :shipit: complete! 1 of 0 LGTMs obtained (waiting on @asubiotto, @dt, and @tbg)


pkg/config/system.go, line 207 at r3 (raw file):

Previously, tbg (Tobias Grieger) wrote…

or the largest pseudo ID, whichever is larger.

Done.


pkg/config/system.go, line 221 at r3 (raw file):

Previously, tbg (Tobias Grieger) wrote…

and pick either it or maxPseudoID.

Done.


pkg/config/system.go, line 229 at r3 (raw file):

Previously, tbg (Tobias Grieger) wrote…

This can't be intentional, at least not without a comment.

This was a drive-by fix. The code was subtly broken before. We could hit an error while decoding and then overwrite it on the next iteration. I added a test that tests a case that would have previously been handled incorrectly.


pkg/keys/constants.go, line 399 at r4 (raw file):

Previously, tbg (Tobias Grieger) wrote…

Can you revisit the "because the bootstrap process needs to create splits for them"? I think I remember that it does create splits for the tables with these IDs (which don't exist). I think something is fishy here, it's not like this package has any notion of where the split points that correspond to these pseudoIDs are. Probably the comment makes no sense.

I'm not sure I understand the ask. We do create splits for these tables, even though they don't really exist. Do you take issue with the keys package mentioning split points?


pkg/sql/drop_table.go, line 354 at r4 (raw file):

Previously, asubiotto (Alfonso Subiotto Marqués) wrote…

I think so. Based on 7abac2f, this code was added as a sanity check. OOC, how would this inability to get a zone config for a dropped tenant table have manifested itself otherwise? Would the default TTL be applied?

Thanks for confirming. It looks like an inability to get a zone config for a dropped tenant table will now manifest as an error being thrown by updateStatusForGCElements from the call to GetZoneConfigForObject. I don't think there's ever a case where a table wouldn't have an associated zone config though, and David seemed to think this was pointless, so I'm going to remove it.


pkg/sql/zone_config.go, line 159 at r2 (raw file):

Previously, tbg (Tobias Grieger) wrote…

While you're here, explain the retvals?

Done.


pkg/sql/logictest/testdata/logic_test/tenant_unsupported, line 37 at r4 (raw file):

Previously, asubiotto (Alfonso Subiotto Marqués) wrote…

You can probably delete this.

Done.

Copy link
Copy Markdown
Contributor

@asubiotto asubiotto left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:lgtm:

Reviewable status: :shipit: complete! 2 of 0 LGTMs obtained (waiting on @asubiotto, @dt, and @tbg)

@tbg tbg requested review from asubiotto and tbg June 4, 2020 10:10
Copy link
Copy Markdown
Member

@tbg tbg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:lgtm:

Reviewed 47 of 47 files at r5, 4 of 4 files at r6, 5 of 5 files at r7, 3 of 3 files at r8, 44 of 44 files at r9.
Reviewable status: :shipit: complete! 2 of 0 LGTMs obtained (waiting on @asubiotto, @dt, and @tbg)

@nvb nvb force-pushed the nvanbenschoten/tenantSplit branch 2 times, most recently from 18df388 to 56f002c Compare June 4, 2020 17:27
@nvb nvb requested a review from a team as a code owner June 4, 2020 17:27
@nvb nvb force-pushed the nvanbenschoten/tenantSplit branch from 56f002c to f59c561 Compare June 4, 2020 17:57
nvb added 4 commits June 4, 2020 14:35
We don't return nil values from MVCCScan.

Also, a bit of harmless cleanup.
Mostly cleaning up comments. Noticed while re-familiarizing
myself with this package.
This was not behaving correctly, and was breaking things when a
pseudo table had the largest ID below MaxReservedDescID.
@nvb nvb force-pushed the nvanbenschoten/tenantSplit branch from f59c561 to 85e4ac4 Compare June 4, 2020 18:35
Fixes cockroachdb#49318.
Fixes cockroachdb#49445.
Progress towards cockroachdb#48123.
Informs cockroachdb#48774.

This commit introduces a new pseudo object ID in the system tenant's
namespace called "tenants". Like "liveness" and "timeseries" before it,
the pseudo object allows zone configurations to be applied to
pseudo-objects that do not live directly in the system tenant's SQL
keyspace. In this case, the "tenants" zone allows zone configurations to
be set by the system tenant and applied to all other tenants in the
system. There may come a time when we want secondary tenants to have
more control over their zone configurations, but that will require a
much larger change to the zone config structure and UX as a whole.

While making this change, we rationalize the rest of zone configuration
handling and how it relates to multi-tenancy. Now that secondary tenant
ranges have a zone config to call their own, we can make sense of calls
from KV into the zone configuration infrastructure. We gate off calls
that don't make sense for secondary tenants and clean up hooks in SQL
that handle zone config manipulation.

All of this works towards a good cause - we eliminate the remaining uses
of `keys.TODOSQLCodec` from `pkg/sql/...` and `pkg/config/...`, bringing
us a big step closer towards being able to remove the placeholder and
close cockroachdb#48123.

This work also reveals that in order to address cockroachdb#48774, we need to be
able to determine split points from the SystemConfig. This makes it very
difficult to split on secondary tenant object (e.g. table) boundaries.
However, it makes it straightforward to split on secondary tenant
keysapce boundaries. This is already what we were thinking (see cockroachdb#47907),
so out both convenience and desire, I expect that we'll follow this up
with a PR that splits Ranges only at secondary tenant boundaries -
placing the overhead of an otherwise empty tenant at only a single Range
and a few KBs of data.
@nvb nvb force-pushed the nvanbenschoten/tenantSplit branch from 85e4ac4 to 6953182 Compare June 4, 2020 20:11
@nvb
Copy link
Copy Markdown
Contributor Author

nvb commented Jun 4, 2020

bors r+

@craig
Copy link
Copy Markdown
Contributor

craig bot commented Jun 4, 2020

Build succeeded

@craig craig bot merged commit 9b50756 into cockroachdb:master Jun 4, 2020
@tbg tbg added the A-multitenancy Related to multi-tenancy label Jun 5, 2020
@nvb nvb deleted the nvanbenschoten/tenantSplit branch June 8, 2020 20:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

A-multitenancy Related to multi-tenancy

Projects

None yet

4 participants