This issue is a followup to https://github.com/cockroachlabs/managed-service/pull/7408
and this comment

Requested by @aaron-crl and @bdarnell

Description of problem

Currently the password code fixes the bcrypt cost to 10 used when generating/storing new password hashes server-side.
The scram PR #74301 follows this model and hardcodes 4096 as the pkbf2 iteration count.

Both values are very low and have become inadequate as of 2022.
They are meant to be increased over time has hardware performance increases.

@bdarnell recalls a heuristic that the bcrypt cost should be increased by 1 each year; and that for pkbf2 / sha256, folk were already using 64000 iterations a few years ago.

CockroachDB should provide some mechanism to evolve the cost function over time

Design ideas

A simple design could be to provide a cluster setting that controls the cost used for new passwords.
For long-lived CockroachDB deployments, we could ship increases to the default value of the cluster setting in patch releases.

A question then arises of what to do with existing passwords, created with a previous (lower) iteration count.

For this, we can observe that increasing the iteration count is an associative operation: iter(iter(H, x), y) = iter(H, x+y)
This means that we can increase the iteration count for an already-computed hash by simply further iterating the hash function on the hash that's stored already.

This helps us define an automatic upgrade for existing passwords: a process that iterates over system.users.hashedPassword and further hashes up to the value indicated by the current target cost.

Design constraint: fixed iteration counts

We have offered operators the ability to pre-hash passwords and store pre-computed hashes.

We should assume that operators are interested to control the cost function. For example, in the CC control plane we generate hashes with a lower iteration count, because we control the entropy of the generated password (see https://github.com/cockroachlabs/managed-service/pull/7408).

When the operator controls the iteration count, we do not wish to automatically update it inside CockroachDB.

Design sketch

• a cluster setting sets the desired target cost.
• the default value of that cluster setting is increased in patch releases over time.
• when a new password is generated from plaintext server-side, we use the cluster setting to select the cost.
• we start storing, in the system.users.hashedPassword column using the format prefix, a bit that indicates whether the cost was auto-selected by crdb or was manually set by an operator.
• (optionally, see comments below) we modify the CC control plane (extending this PR) to use that bit to force the cost function to remain fixed.
• periodically, we scan system.users and upgrade the non-fixed hashes to the configured target cost in the cluster setting.

Epic CRDB-5349