Upgrades were taking a reference to the job running them. Once upon a
time, this was allegedly used by some upgrade in order to checkpoint its
progress. Nobody uses it currently, and this patch removes it. We can
add it back if we need it again.
This reference was in my way because in cockroachdb#91627 I'm adding a testing mode
where upgrades do not run in jobs.

Release note: None

91929: startupmigrations: remove useless guard in migration r=andreimatei a=andreimatei

The migration writing the initial value of the cluster version had a conditional about whether it's performing the migration for the system tenant. The migration only runs as the system tenant, courtesy of the clusterWide setting [1]. This patch this replaces the conditional with an assertion.

[1] https://github.com/cockroachdb/cockroach/blob/fd42d376fb5ec42e9d9a30fcdcb210a8a79ef0e2/pkg/startupmigrations/migrations.go#L108

Release note: None
Epic: None

91931: startupmigrations: remove no-op SET CLUSTER SETTING version r=andreimatei a=andreimatei

The migration writing the initial value of the cluster version setting was also performing a `SET CLUSTER SETTING version = v`, where v was the version is had just written in the settings table. This statement was a no-op, because it amounted to an update from v to v [1]. This patch removes the statement.

[1] https://github.com/cockroachdb/cockroach/blob/f87728fee6d16e2a352ac76e7ae9225930032ce6/pkg/upgrade/upgrademanager/manager.go#L167-L171

91998: upgrade: minor cleanups r=andreimatei a=andreimatei

See individual commits.
Extracted from #91627

Release note: None
Epic: None

Co-authored-by: Andrei Matei <andrei@cockroachlabs.com>

This patch removes a permanent upgrade that was granting the CREATELOGIN
role option to users that had the CREATEROLE option. This upgrade was
introduced in 20.2, meant to grant the then-new CREATELOGIN option to
users created in 20.1 and prior that had the CREATEROLE option
(CREATELOGIN was being split from CREATEROLE).

This code stayed around as a startupmigration since it was introduced,
even though it didn't make sense for it to stay around after 20.2.
Technically, I think this startup migration should have had the flag
`IncludedInBootstrap=v20.2`, since we don't want it to run for clusters
created at or after 20.2; this migration is not idempotent in the
general sense, since it potentially picks up new, unintended users when
it runs. Since 22.2, this migration would fail to run on anything but an
empty system.role_options table because it would attempt to put NULLs
into a non-nullable column. This was all benign since the
startupmigrations had protection in place, preventing them for running a
2nd time after a completed attempt. So, for upgraded cluster, the
migration only when the first 20.2 node came up, and for new clusters it
would be a no-op since the system.role_options table starts empty.

This migration became problematic in cockroachdb#91627 when I've turned the
startupmigrations into upgrades. These upgrades run once when upgrading
to 23.1; I'm relying on their idempotence.

Fixes cockroachdb#92230
Fixes cockroachdb#92371
Fixes cockroachdb#92569

Release note: None

Some of the permanent upgrades (opting into telemetry and initializing
the cluster secret) were not idempotent; this patch makes them be. In
particular, this is important since these upgrades will run on 23.1
cluster that are being upgraded from 22.2 as they've been turned from
startupmigrations to upgrades in cockroachdb#91627. For the telemetry one, there's
no "natural" way to make it idempotent, so I've added a check for the
old startupmigration key.

Release note: None
Epic: None

92289: catalog,tabledesc: add & adopt Constraint subtypes r=Xiang-Gu a=postamar

These new interfaces help encapsulate the descriptor protobufs for table
constraints:
  - CheckConstraint,
  - ForeignKeyConstraint,
  - UniqueWithoutIndexConstraint,
  - UniqueWithIndexConstraint.

These are the leaves of the Constraint type tree, and Constraint is the
interface at the root of it. The changes in this commit mimic what was
done for Column & Index, which respectively wrap descpb.ColumnDescriptor
and descpb.IndexDescriptor.

These new constraint interfaces are adopted liberally throughout the
code base, but not completely: many references to descpb-protobufs still
need to be replaced.

This commit also removes the somewhat confusing concept of "active"
versus "inactive" constraint. Not only was it confusing to me, evidently
it was also confusing to other contributors considering how several mild
bugs and inconsistencies made their way into the code. This commit
replaces this concept with that of an "enforced" constraint:
  - A constraint is enforced if it applies to data written to the table,
    regardless of whether it has been validated on the data already in
    the table prior to the constraint coming into existence.

The implementation is somewhat awkward for constraints in that the same
constraint descriptor can be featured twice inside
a descpb.TableDescriptor, in its active constraint slice (such as
Checks, etc.) and also in the Mutations slice. In these cases the
interface wraps the non-mutation constraint descriptor protobuf, with no
observed changes to the database's behavior.

This commit required alterations to the table descriptor's
post-deserialization changes. The change which assigns constraint IDs
to table descriptors which don't have themhad some subtle bugs which went
unnoticed until this commit forced a heavier reliance on these fields.
This commit also adds a new change which ensures that a check
constraint's ColumnIDs slice is populated based on the columns
referenced in its expression, replacing ancient code which predates the
concept of post-deserialization changes and which would compute this
lazily.

As a more general observation: it appears that the handling of adding
and dropping constraints following other schema changes was mildly
inconsistent and this commit tries to improve this. For instance, it was
possible to drop a column which had been set as NOT NULL in the same
transaction, but not do the same for any other kind of constraint. Also
it was possible to reuse the name of a dropping constraint, despite this
constraint still being enforced. The new behavior is more strict, yet
this should be barely noticeable by the user in most practical cases:
it's rare that one wants to add a constraint referencing a column and
also drop that column in the same transaction, for instance.

Fixes #91918.

Release note: None

92597: upgrades: remove upgrade granting CREATELOGIN role opt r=andreimatei a=andreimatei

This patch removes a permanent upgrade that was granting the CREATELOGIN role option to users that had the CREATEROLE option. This upgrade was introduced in 20.2, meant to grant the then-new CREATELOGIN option to users created in 20.1 and prior that had the CREATEROLE option (CREATELOGIN was being split from CREATEROLE).

This code stayed around as a startupmigration since it was introduced, even though it didn't make sense for it to stay around after 20.2. Technically, I think this startup migration should have had the flag `IncludedInBootstrap=v20.2`, since we don't want it to run for clusters created at or after 20.2; this migration is not idempotent in the general sense, since it potentially picks up new, unintended users when it runs. Since 22.2, this migration would fail to run on anything but an empty system.role_options table because it would attempt to put NULLs into a non-nullable column. This was all benign since the startupmigrations had protection in place, preventing them for running a 2nd time after a completed attempt. So, for upgraded cluster, the migration only when the first 20.2 node came up, and for new clusters it would be a no-op since the system.role_options table starts empty.

This migration became problematic in #91627 when I've turned the startupmigrations into upgrades. These upgrades run once when upgrading to 23.1; I'm relying on their idempotence.

Fixes #92230
Fixes #92371
Fixes #92569

Release note: None
Epic: None

Co-authored-by: Marius Posta <marius@cockroachlabs.com>
Co-authored-by: Andrei Matei <andrei@cockroachlabs.com>

Some of the permanent upgrades (opting into telemetry and initializing
the cluster secret) were not idempotent; this patch makes them be. In
particular, this is important since these upgrades will run on 23.1
cluster that are being upgraded from 22.2 as they've been turned from
startupmigrations to upgrades in cockroachdb#91627. For the telemetry one, there's
no "natural" way to make it idempotent, so I've added a check for the
old startupmigration key.

Release note: None
Epic: None

90842: sql: add tsvector and tsquery types to SQL r=jordanlewis a=jordanlewis

Updates #41288.

Broken off from #83736

This PR adds the tsvector and tsquery types and pgwire encodings to sql, and makes them available for use using the string::tsvector and string::tsquery casts. It also implements the in-memory evaluation of `@@` (the tsquery match operator) by delegating to the tsearch.eval package.

Release note (sql change): implement in memory handling for the tsquery and tsvector Postgres datatypes, which provide text search capabilities. See https://www.postgresql.org/docs/current/datatype-textsearch.html for more details.

Epic: None

92611: upgrades: make a couple of permanent upgrades idempotent r=andreimatei a=andreimatei

Some of the permanent upgrades (opting into telemetry and initializing the cluster secret) were not idempotent; this patch makes them be. In particular, this is important since these upgrades will run on 23.1 cluster that are being upgraded from 22.2 as they've been turned from startupmigrations to upgrades in #91627. For the telemetry one, there's no "natural" way to make it idempotent, so I've added a check for the old startupmigration key.

Release note: None
Epic: None

92620: upgrademanager: clarify job running code r=andreimatei a=andreimatei

This patch makes the code clearer around running upgrade in jobs. Before this patch, the upgrade manager was looking for existing jobs corresponding to upgrades and, if it found them, it called jobsRegistry.Run() on them - just like it did for newly-created jobs. The behavior of Run() for jobs that are not already claimed by the registry is a bit under-specified. I believe it ended up printing an error [1] and then correctly waiting for the job to finish.

This patch no longer calls Run() for existing jobs; instead, it calls WaitForJobs(), which is more suggestive.

[1] https://github.com/cockroachdb/cockroach/blob/b2a6b80920324bd6b31cba9a6f622961979de600/pkg/jobs/adopt.go#L255

Release note: None
Epic: None

Co-authored-by: Jordan Lewis <jordanthelewis@gmail.com>
Co-authored-by: Andrei Matei <andrei@cockroachlabs.com>

This patch makes the "primordial" permanent upgrades not run in clusters
being upgraded from 22.2 or lower.

Startup migrations were recently replaced with "permanent upgrades", in cockroachdb#91627.
The permanent new upgrades were running on node startup regardless of
whether it was a "new cluster" or an existing one - i.e. regardless of
whether the previous startupmigrations had also run, on the argument
that startupmigrations and upgrades are supposed to be idempotent
anyway. But, alas, there's idempotency and then there's idempotency:
startupmigrations/upgrades are supposed to be OK with running multiple
times at the same BinaryVersion in a "new cluster", but not necessarily
across BinaryVersions and in a cluster that has been in use. There are
different things that can go wrong when one of these migrations runs
multiple times in these general conditions, because the migration
might makes implicit assumptions about the schema and data of the
cluster it's running in:
1. A migration might assume that all the data in a system table was
   created before a CRDB version changed some semantics. For example,
   the migration deleted in cockroachdb#92597 was assuming that `CREATEROLE` role
   option had a certain meaning for all the users who have it and was doing
   a backfill across all these users. Running this migration again at an
   arbitrary later point would affect unintended new users.
2. A migration assumes the schema of system tables it operates on. As
   this schema (which is part of the bootstrap image for new cluster)
   changes with new versions, the code of the upgrade changes also. This
   means that the upgrade is no longer suitable for running against
   clusters that don't have the upgraded schema - i.e. clusters that have
   not yet been upgraded to the latest schema. This is a real problem
   because we're adding a column to system.users, and the addRootUser
   upgrade can no longer run against 22.2 clusters.

This patch guards all upgrades on checking that the old corresponding
startupmigration has not run, thus enabling the mentioned system schema
change in 23.1.

Release note: None
Epic: None

This patch makes the "primordial" permanent upgrades not run in clusters
being upgraded from 22.2 or lower.

Startup migrations were recently replaced with "permanent upgrades", in cockroachdb#91627.
The permanent new upgrades were running on node startup regardless of
whether it was a "new cluster" or an existing one - i.e. regardless of
whether the previous startupmigrations had also run, on the argument
that startupmigrations and upgrades are supposed to be idempotent
anyway. But, alas, there's idempotency and then there's idempotency:
startupmigrations/upgrades are supposed to be OK with running multiple
times at the same BinaryVersion in a "new cluster", but not necessarily
across BinaryVersions and in a cluster that has been in use. There are
different things that can go wrong when one of these migrations runs
multiple times in these general conditions, because the migration
might makes implicit assumptions about the schema and data of the
cluster it's running in:
1. A migration might assume that all the data in a system table was
   created before a CRDB version changed some semantics. For example,
   the migration deleted in cockroachdb#92597 was assuming that `CREATEROLE` role
   option had a certain meaning for all the users who have it and was doing
   a backfill across all these users. Running this migration again at an
   arbitrary later point would affect unintended new users.
2. A migration assumes the schema of system tables it operates on. As
   this schema (which is part of the bootstrap image for new cluster)
   changes with new versions, the code of the upgrade changes also. This
   means that the upgrade is no longer suitable for running against
   clusters that don't have the upgraded schema - i.e. clusters that have
   not yet been upgraded to the latest schema. This is a real problem
   because we're adding a column to system.users, and the addRootUser
   upgrade can no longer run against 22.2 clusters.

This patch guards all upgrades on checking that the old corresponding
startupmigration has not run, thus enabling the mentioned system schema
change in 23.1.

Release note: None
Epic: None

92363: sql,streamingest: introduce ALTER TENANT ... PAUSE/RESUME REPLICATION r=lidorcarmel a=adityamaru

This change introduces:
`ALTER TENANT <name> RESUME REPLICATION`
`ALTER TENANT <name> PAUSE REPLICATION`

These statements can be used to pause and resume the replication job if the tenant is being replicated into.

Fixes: #91246

Release note: None

93071: upgrade: don't blindly run permanent migrations r=andreimatei a=andreimatei

This patch makes the "primordial" permanent upgrades not run in clusters
being upgraded from 22.2 or lower.

Startup migrations were recently replaced with "permanent upgrades", in #91627.
The permanent new upgrades were running on node startup regardless of
whether it was a "new cluster" or an existing one - i.e. regardless of
whether the previous startupmigrations had also run, on the argument
that startupmigrations and upgrades are supposed to be idempotent
anyway. But, alas, there's idempotency and then there's idempotency:
startupmigrations/upgrades are supposed to be OK with running multiple
times at the same BinaryVersion in a "new cluster", but not necessarily
across BinaryVersions and in a cluster that has been in use. There are
different things that can go wrong when one of these migrations runs
multiple times in these general conditions, because the migration
might makes implicit assumptions about the schema and data of the
cluster it's running in:
1. A migration might assume that all the data in a system table was
   created before a CRDB version changed some semantics. For example,
   the migration deleted in #92597 was assuming that `CREATEROLE` role
   option had a certain meaning for all the users who have it and was doing
   a backfill across all these users. Running this migration again at an
   arbitrary later point would affect unintended new users.
2. A migration assumes the schema of system tables it operates on. As
   this schema (which is part of the bootstrap image for new cluster)
   changes with new versions, the code of the upgrade changes also. This
   means that the upgrade is no longer suitable for running against
   clusters that don't have the upgraded schema - i.e. clusters that have
   not yet been upgraded to the latest schema. This is a real problem
   because we're adding a column to system.users, and the addRootUser
   upgrade can no longer run against 22.2 clusters.

This patch guards all upgrades on checking that the old corresponding
startupmigration has not run, thus enabling the mentioned system schema
change in 23.1.

Release note: None
Epic: None

Co-authored-by: adityamaru <adityamaru@gmail.com>
Co-authored-by: Andrei Matei <andrei@cockroachlabs.com>

This is no longer referenced since cockroachdb#91627.

Epic: none

Release note: None

94239: loqrecovery: use captured meta range content for LOQ plans r=erikgrinaker a=aliher1911

Note: only last commit belongs to this PR. Will update description once #93157 lands.

Previously loss of quorum recovery planner was using local replica info collected from all nodes to find source of truth for replicas that lost quorum.
With online approach local info snapshots don't have atomicity. This could cause planner to fail if available replicas are caught in different states on different nodes.
This commit adds alternative planning approach when range metadata is available. Instead of fixing individual replicas that can't make progress it finds ranges that can't make progress from metadata using descriptors and updates their replicas to recover from loss of quorum.
This commit also adds replica collection stage as a part of make-plan command itself. To invoke collection from a cluster instead of using files one needs to provide --host and other standard cluster connection related flags (--cert-dir, --insecure etc.) as appropriate.

Example command output for a local cluster with 3 out of 5 nodes surrvivng looks like:
```
~/tmp ❯❯❯ cockroach debug recover make-plan --insecure --host 127.0.0.1:26257 >recover-plan.json
Nodes scanned:           3
Total replicas analyzed: 228
Ranges without quorum:   15
Discarded live replicas: 0

Proposed changes:
  range r4:/System/tsd updating replica (n2,s2):3 to (n2,s2):15. Discarding available replicas: [], discarding dead replicas: [(n5,s5):4,(n4,s4):2].
  range r80:/Table/106/1 updating replica (n1,s1):1 to (n1,s1):14. Discarding available replicas: [], discarding dead replicas: [(n5,s5):3,(n4,s4):2].
  range r87:/Table/106/1/"paris"/"\xcc\xcc\xcc\xcc\xcc\xcc@\x00\x80\x00\x00\x00\x00\x00\x00(" updating replica (n1,s1):1 to (n1,s1):14. Discarding available replicas: [], discarding dead replicas: [(n5,s5):3,(n4,s4):2].
  range r88:/Table/106/1/"seattle"/"ffffffH\x00\x80\x00\x00\x00\x00\x00\x00\x14" updating replica (n3,s3):3 to (n3,s3):15. Discarding available replicas: [], discarding dead replicas: [(n5,s5):4,(n4,s4):2].
  range r105:/Table/106/1/"washington dc"/"L\xcc\xcc\xcc\xcc\xccL\x00\x80\x00\x00\x00\x00\x00\x00\x0f" updating replica (n3,s3):3 to (n3,s3):14. Discarding available replicas: [], discarding dead replicas: [(n5,s5):1,(n4,s4):2].
  range r98:/Table/107/1/"boston"/"333333D\x00\x80\x00\x00\x00\x00\x00\x00\x03" updating replica (n2,s2):3 to (n2,s2):15. Discarding available replicas: [], discarding dead replicas: [(n5,s5):4,(n4,s4):2].
  range r95:/Table/107/1/"seattle"/"ffffffH\x00\x80\x00\x00\x00\x00\x00\x00\x06" updating replica (n3,s3):2 to (n3,s3):15. Discarding available replicas: [], discarding dead replicas: [(n4,s4):4,(n5,s5):3].
  range r125:/Table/107/1/"washington dc"/"DDDDDDD\x00\x80\x00\x00\x00\x00\x00\x00\x04" updating replica (n3,s3):2 to (n3,s3):14. Discarding available replicas: [], discarding dead replicas: [(n4,s4):1,(n5,s5):3].
  range r115:/Table/108/1/"boston"/"8Q\xeb\x85\x1e\xb8B\x00\x80\x00\x00\x00\x00\x00\x00n" updating replica (n2,s2):3 to (n2,s2):15. Discarding available replicas: [], discarding dead replicas: [(n5,s5):4,(n4,s4):2].
  range r104:/Table/108/1/"new york"/"\x1c(\xf5\u008f\\I\x00\x80\x00\x00\x00\x00\x00\x007" updating replica (n2,s2):2 to (n2,s2):15. Discarding available replicas: [], discarding dead replicas: [(n5,s5):4,(n4,s4):3].
  range r102:/Table/108/1/"seattle"/"p\xa3\xd7\n=pD\x00\x80\x00\x00\x00\x00\x00\x00\xdc" updating replica (n3,s3):2 to (n3,s3):15. Discarding available replicas: [], discarding dead replicas: [(n4,s4):4,(n5,s5):3].
  range r126:/Table/108/1/"washington dc"/"Tz\xe1G\xae\x14L\x00\x80\x00\x00\x00\x00\x00\x00\xa5" updating replica (n3,s3):2 to (n3,s3):14. Discarding available replicas: [], discarding dead replicas: [(n4,s4):1,(n5,s5):3].
  range r86:/Table/108/3 updating replica (n1,s1):1 to (n1,s1):14. Discarding available replicas: [], discarding dead replicas: [(n4,s4):3,(n5,s5):2].
  range r59:/Table/109/1 updating replica (n2,s2):3 to (n2,s2):15. Discarding available replicas: [], discarding dead replicas: [(n5,s5):4,(n4,s4):2].
  range r65:/Table/111/1 updating replica (n3,s3):3 to (n3,s3):15. Discarding available replicas: [], discarding dead replicas: [(n5,s5):4,(n4,s4):2].

Discovered dead nodes would be marked as decommissioned:
  n4, n5


Proceed with plan creation [y/N] y
Plan created.
To stage recovery application in half-online mode invoke:

'cockroach debug recover apply-plan  --host=127.0.0.1:26257 --insecure=true <plan file>'

Alternatively distribute plan to below nodes and invoke 'debug recover apply-plan --store=<store-dir> <plan file>' on:
- node n2, store(s) s2
- node n1, store(s) s1
- node n3, store(s) s3
```

Release note: None

Fixes: #93038
Fixes: #93046

94948: changefeedccl: give notice when metrics_label set without child_metrics r=samiskin a=samiskin

Resolves #75682

Surfaces a notice of
```
server.child_metrics.enabled is set to false, metrics will only be published to the '<scope>' label when it is set to true"
```
When child_metrics setting isn't enabled during changefeed creation

Release note (enterprise change): Changefeeds created/altered with a metrics_label set while server.child_metrics.enabled is false will now provide the user a notice upon creation.

95009: tree: fix panic when encoding tuple r=rafiss a=rafiss

fixes #95008

This adds a bounds check to avoid a panic.

Release note (bug fix): Fixed a crash that could happen when formatting a tuple with an unknown type.

95294: sql: make pg_description aware of builtin function descriptions r=rafiss,msirek a=knz

Epic: CRDB-23454
Fixes #95292.
Needed for #88061. 

First commit from #95289.

This also extends the completion rules to properly handle
functions in multiple namespaces.

Release note (bug fix): `pg_catalog.pg_description` and `pg_catalog.obj_description()` are now able to retrieve the descriptive help for built-in functions.

95356: server: remove unused migrationExecutor r=ajwerner a=ajwerner

This is no longer referenced since #91627.

Epic: none

Release note: None

Co-authored-by: Oleg Afanasyev <oleg@cockroachlabs.com>
Co-authored-by: Shiranka Miskin <shiranka.miskin@gmail.com>
Co-authored-by: Rafi Shamim <rafi@cockroachlabs.com>
Co-authored-by: Raphael 'kena' Poss <knz@thaumogen.net>
Co-authored-by: Andrew Werner <awerner32@gmail.com>

This partially reverts commit 129d62a.

It also fixes the roachtest that was added in the previous commit.

In PR cockroachdb#119142, the permanent upgrades were flattened so that they
all have a version of 0.0-x. That means that the last element of
the permanentUpgrades slice has a version of 0.0-4. Clusters that were
bootstrapped with a version from v23.1.0 or later will have rows in
system.migrations for versions 0.0-x, since that is the version where
the permanent upgrades framework was introduced (in PR cockroachdb#91627).
However, clusters that are were bootstrapped from a version
earlier than v23.1 will not yet have these rows in system.migrations for
versions 0.0-x, and those migrations will only be present under the
legacy startupmigrations key.

Release note (bug fix): Fixed a bug that could cause the password for
the root user to be deleted while upgrading to v24.1. This bug only
affects clusters that were initially created with a version of v22.2
or earlier.

The same bug could also cause the `defaultdb` and `postgres` databases
to be recreated during the upgrade to v24.1 if they had been previously
deleted.