Patch #1 caused several problems in mysql-trunk related to ndbinfo
initialization and upgrade, including the failure of the test
ndb_76_inplace_upgrade and the failure of all NDB MTR tests in Pushbuild
on Windows. This patch fixes these issues, including fixes for
bug#33726826 and bug#33730799.

In ndbinfo, revert the removal of ndb$blocks and ndb$index_stats and the
change of blocks and index_stats from views to tables.

Improve the ndbinfo schema upgrade & initialization logic to better handle
such a change in the future. This logic now runs in two passes: first it
drops the known tables and views from current and previous versions, then
it creates the tables and views for the current version.

Add a new class method NdbDictionary::printColumnTypeDescription(). This
is needed for the ndbinfo.columns table in patch #2 but was missing from
patch #1. Add boilerplate index lookup initialization code that was
also missing.

Fix ndbinfo prefix determination on Windows.

Change-Id: I422856bcad4baf5ae9b14c1e3a1f2871bd6c5f59

* PROBLEM

The test "ndb.ndb_bug17624736" was constantly failing in
[daily|weekly]-8.0-cluster branches in PB2, whether on `ndb-ps` or
`ndb-default-big` profile test runs. The high-level reason for the
failure was the installation of a duplicate entry in the Data
Dictionary in respect to the `engine`-`se_private_id` pair, even when
the previous table definition should have been dropped.

* LOW-LEVEL EXPLANATION

NDB reuses the least available ID for the dictionary table ID. The ID
is then used by the NDB plugin to install as SE private ID field of
the MySQL Data Dictionary table definition. If a problem occurs during
the synchronization of NDB table definitions in the Data
Dictionary (i.e., a previous definition was not successfully removed),
then an attempt to install a table using an already installed SE
private ID can occur. If that ID was inadvertedly cached as `missing`,
then the function `acquire_uncached_table_by_se_private_id` will
return fast without retrieving the table definition. Therefore, the
old table definition on that ID cannot be retrieved ever for that Data
Dictionary client instance, the new one won't be installed, and errors
will be raised.

* SOLUTION

For NDB plugin to query a table definition, using the SE private
ID (without causing a missing entry to be cached forever for that
client instance), this patch adds a flag argument to the function to
allow the caller to request it to skip the fast cache.

Change-Id: I45eef594ee544000fe6b30b86977e5e91155dc80

-- Patch #1: Persist secondary load information --

Problem:
We need a way of knowing which tables were loaded to HeatWave after
MySQL restarts due to a crash or a planned shutdown.

Solution:
Add a new "secondary_load" flag to the `options` column of mysql.tables.
This flag is toggled after a successful secondary load or unload. The
information about this flag is also reflected in
INFORMATION_SCHEMA.TABLES.CREATE_OPTIONS.

-- Patch #2 --

The second patch in this worklog triggers the table reload from InnoDB
after MySQL restart.

The recovery framework recognizes that the system restarted by checking
whether tables are present in the Global State. If there are no tables
present, the framework will access the Data Dictionary and find which
tables were loaded before the restart.

This patch introduces the "Data Dictionary Worker" - a MySQL service
recovery worker whose task is to query the INFORMATION_SCHEMA.TABLES
table from a separate thread and find all tables whose secondary_load
flag is set to 1.

All tables that were found in the Data Dictionary will be appended to
the list of tables that have to be reloaded by the framework from
InnoDB.

If an error occurs during restart recovery we will not mark the recovery
as failed. This is done because the types of failures that can occur
when the tables are reloaded after a restart are less critical compared
to previously existing recovery situations. Additionally, this code will
soon have to be adapted for the next worklog in this area so we are
proceeding with the simplest solution that makes sense.

A Global Context variable m_globalStateEmpty is added which indicates
whether the Global State should be recovered from an external source.

-- Patch #3 --

This patch adds the "rapid_reload_on_restart" system variable. This
variable is used to control whether tables should be reloaded after a
restart of mysqld or the HeatWave plugin. This variable is persistable
(i.e., SET PERSIST RAPID_RELOAD_ON_RESTART = TRUE/FALSE).

The default value of this variable is set to false.

The variable can be modified in OFF, IDLE, and SUSPENDED states.

-- Patch #4 --

This patch refactors the recovery code by removing all recovery-related
code from ha_rpd.cc and moving it to separate files:

  - ha_rpd_session_factory.h/cc:
  These files contain the MySQLAdminSessionFactory class, which is used
to create admin sessions in separate threads that can be used to issue
SQL queries.

  - ha_rpd_recovery.h/cc:
  These files contain the MySQLServiceRecoveryWorker,
MySQLServiceRecoveryJob and ObjectStoreRecoveryJob classes which were
previously defined in ha_rpd.cc. This file also contains a function that
creates the RecoveryWorkerFactory object. This object is passed to the
constructor of the Recovery Framework and is used to communicate with
the other section of the code located in rpdrecoveryfwk.h/cc.

This patch also renames rpdrecvryfwk to rpdrecoveryfwk for better
readability.

The include relationship between the files is shown on the following
diagram:

        rpdrecoveryfwk.h◄──────────────rpdrecoveryfwk.cc
            ▲    ▲
            │    │
            │    │
            │    └──────────────────────────┐
            │                               │
        ha_rpd_recovery.h◄─────────────ha_rpd_recovery.cc──┐
            ▲                               │           │
            │                               │           │
            │                               │           │
            │                               ▼           │
        ha_rpd.cc───────────────────────►ha_rpd.h       │
                                            ▲           │
                                            │           │
            ┌───────────────────────────────┘           │
            │                                           ▼
    ha_rpd_session_factory.cc──────►ha_rpd_session_factory.h

Other changes:
  - In agreement with Control Plane, the external Global State is now
  invalidated during recovery framework startup if:
    1) Recovery framework recognizes that it should load the Global
    State from an external source AND,
    2) rapid_reload_on_restart is set to OFF.

  - Addressed review comments for Patch #3, rapid_reload_on_restart is
  now also settable while plugin is ON.

  - Provide a single entry point for processing external Global State
  before starting the recovery framework loop.

  - Change when the Data Dictionary is read. Now we will no longer wait
  for the HeatWave nodes to connect before querying the Data Dictionary.
  We will query it when the recovery framework starts, before accepting
  any actions in the recovery loop.

  - Change the reload flow by inserting fake global state entries for
  tables that need to be reloaded instead of manually adding them to a
  list of tables scheduled for reload. This method will be used for the
  next phase where we will recover from Object Storage so both recovery
  methods will now follow the same flow.

  - Update secondary_load_dd_flag added in Patch #1.

  - Increase timeout in wait_for_server_bootup to 300s to account for
  long MySQL version upgrades.

  - Add reload_on_restart and reload_on_restart_dbg tests to the rapid
  suite.

  - Add PLUGIN_VAR_PERSIST_AS_READ_ONLY flag to "rapid_net_orma_port"
  and "rapid_reload_on_restart" definitions, enabling their
  initialization from persisted values along with "rapid_bootstrap" when
  it is persisted as ON.

  - Fix numerous clang-tidy warnings in recovery code.

  - Prevent suspended_basic and secondary_load_dd_flag tests to run on
  ASAN builds due to an existing issue when reinstalling the RAPID
  plugin.

-- Bug#33752387 --

Problem:
A shutdown of MySQL causes a crash in queries fired by DD worker.

Solution:
Prevent MySQL from killing DD worker's queries by instantiating a
DD_kill_immunizer before the queries are fired.

-- Patch #5 --

Problem:
A table can be loaded before the DD Worker queries the Data Dictionary.
This means that table will be wrongly processed as part of the external
global state.

Solution:
If the table is present in the current in-memory global state we will
not consider it as part of the external global state and we will not
process it by the recovery framework.

-- Bug#34197659 --

Problem:
If a table reload after restart causes OOM the cluster will go into
RECOVERYFAILED state.

Solution:
Recognize when the tables are being reloaded after restart and do not
move the cluster into RECOVERYFAILED. In that case only the current
reload will fail and the reload for other tables will be attempted.

Change-Id: Ic0c2a763bc338ea1ae6a7121ff3d55b456271bf0

Enh#34350907 - [Nvidia] Allow DDLs when tables are loaded to HeatWave
Bug#34433145 - WL#15129: mysqld crash Assertion `column_count == static_cast<int64_t>(cp_table-
Bug#34446287 - WL#15129: mysqld crash at rapid::data::RapidNetChunkCtx::consolidateEncodingsDic
Bug#34520634 - MYSQLD CRASH : Sql_cmd_secondary_load_unload::mysql_secondary_load_or_unload
Bug#34520630 - Failed Condition: "table_id != InvalidTableId"

Currently, DDL statements such as ALTER TABLE*, RENAME TABLE, and
TRUNCATE TABLE are not allowed if a table has a secondary engine
defined. The statements fail with the following error: "DDLs on a table
with a secondary engine defined are not allowed."

This worklog lifts this restriction for tables whose secondary engine is
RAPID.

A secondary engine hook is called in the beginning (pre-hook) and in the
end (post-hook) of a DDL statement execution. If the DDL statement
succeeds, the post-hook will direct the recovery framework to reload the
table in order to reflect that change in HeatWave.

Currently all DDL statements that were previously disallowed will
trigger a reload. This can be improved in the future by checking whether
the DDL operation has an impact on HeatWave or not. However detecting
all edge-cases in this behavior is not straightforward so this
improvement has been left as a future improvement.

Additionally, if a DDL modifies the table schema in a way that makes it
incompatible with HeatWave (e.g., dropping a primary key column) the
reload will fail silently. There is no easy way to recognize whether the
table schema will become incompatible with HeatWave in a pre-hook.

List of changes:
  1) [MySQL] Add new HTON_SECONDARY_ENGINE_SUPPORTS_DDL flag to indicate
whether a secondary engine supports DDLs.
  2) [MySQL] Add RAII hooks for RENAME TABLE and TRUNCATE TABLE, modeled
on the ALTER TABLE hook.
  3) Define HeatWave hooks for ALTER TABLE, RENAME TABLE, and TRUNCATE
TABLE statements.
  4) If a table reload is necessary, trigger it by marking the table as
stale (WL#14914).
  4) Move all change propagation & DDL hooks to ha_rpd_hooks.cc.
  5) Adjust existing tests to support table reload upon DDL execution.
  6) Extract code related to RapidOpSyncCtx in ha_rpd_sync_ctx.cc, and
the PluginState enum to ha_rpd_fsm.h.

* Note: ALTER TABLE statements related to secondary engine setting and
loading were allowed before:
    - ALTER TABLE <TABLE> SECONDARY_UNLOAD,
    - ALTER TABLE SECONDARY_ENGINE = NULL.

-- Bug#34433145 --
-- Bug#34446287 --

--Problem #1--
Crashes in Change Propagation when the CP thread tries to apply DMLs of
tables with new schema to the not-yet-reloaded table in HeatWave.

--Solution #1--
Remove table from Change Propagation before marking it as stale and
revert the original change from rpd_binlog_parser.cc where we were
checking if the table was stale before continuing with binlog parsing.
The original change is no longer necessary since the table is removed
from CP before being marked as stale.

--Problem #2--
In case of a failed reload, tables are not removed from Global State.

--Solution #2--
Keep track of whether the table was reloaded because it was marked as
STALE. In that case we do not want the Recovery Framework to retry the
reload and therefore we can remove the table from the Global State.

-- Bug#34520634 --

Problem:
Allowing the change of primary engine for tables with a defined
secondary engine hits an assertion in mysql_secondary_load_or_unload().

Example:
    CREATE TABLE t1 (col1 INT PRIMARY KEY) SECONDARY_ENGINE = RAPID;
    ALTER TABLE t1 ENGINE = BLACKHOLE;
    ALTER TABLE t1 SECONDARY_LOAD; <- assertion hit here

Solution:
Disallow changing the primary engine for tables with a defined secondary
engine.

-- Bug#34520630 --

Problem:
A debug assert is being hit in rapid_gs_is_table_reloading_from_stale
because the table was dropped in the meantime.

Solution:
Instead of asserting, just return false if table is not present in the
Global State.

This patch also changes rapid_gs_is_table_reloading_from_stale to a more
specific check (inlined the logic in load_table()). This check now also
covers the case when a table was dropped/unloaded before the Recovery
Framework marked it as INRECOVERY. In that case, if the reload fails we
should not have an entry for that table in the Global State.

The patch also adjusts dict_types MTR test, where we no longer expect
for tables to be in UNAVAIL state after a failed reload. Additionally,
recovery2_ddls.test is adjusted to not try to offload queries running on
Performance Schema.

Change-Id: I6ee390b1f418120925f5359d5e9365f0a6a415ee

… Signal (get_store_key at sql/sql_select.cc:2383)

These are two related but distinct problems manifested in the shrinkage
of key definitions for derived tables or common table expressions,
implemented in JOIN::finalize_derived_keys().

The problem in Bug#34572040 is that we have two references to one CTE,
each with a valid key definition. The function will first loop over
the first reference (cte_a) and move its used key from position 0 to
position 1. Next, it will attempt to move the key for the second
reference (cte_b) from position 4 to position 2.
However, for each iteration, the function will calculate used key
information. On the first iteration, the values are correct, but since
key value #1 has been moved into position #0, the old information is
invalid and provides wrong information. The problem is thus that for
subsequent iterations we read data that has been invalidated by earlier
key moves.

The best solution to the problem is to move the keys for all references
to the CTE in one operation. This way, we can calculate used keys
information safely, before any move operation has been performed.

The problem in Bug#34634469 is also related to having more than one
reference to a CTE, but in this case the first reference (ref_3) has
a key in position 5 which is moved to position 0, and the second
reference (ref_4) has a key in position 3 that is moved to position 1.

However, the key parts of the first key will overlap with the key parts
of the second key after the first move, thus invalidating the key
structure during the copy. The actual problem is that we move
a higher-numbered key (5) before a lower-numbered key (3), which in
this case makes it impossible to find an empty space for the moved key.

The solution to this problem is to ensure that keys are moved in
increasing key order.

The patch changes the algorithm as follows:

- When identifying a derived table/common table expression, ensure
  to move all its keys in one operation (at least those references
  from the same query block).

- First, collect information about all key uses: hash key,
  unique index keys and actual key references.
  For the key references, also populate a mapping array that enumerates
  table references with key references in order of increasing
  key number.
  Also clear used key information for references that do not use keys.

- For each table reference with a key reference in increasing key order,
  move the used key into the lowest available position. This will ensure
  that used entries are never overwritten.

- When all table references have been processed, remove unused key
  definitions.

Change-Id: I938099284e34a81886621f6a389f34abc51e78ba

The shadow tables used by ndb_binlog thread when handling data events
have no knowledge about wheter the table is a foreign key parent in NDB,
this is mainly because the NDB dictionary extra metadata for parent
table is not updated when creating the child tables. The information
about being parent table is required in order to properly detect a
transaction conflict while writing changes to the binlog for changes of
parent key tables.

This is fixed by extending the ndb_binlog thread with a metadata cache
that maintains a list of tables in NDB which are foreign key parents.
This makes it possible to update the shadow table's knowledge about
being parent table while handling data events.

Change-Id: Ic551fde3e9460e3668d5aa1ac951ee3bec442cf5

If mysql_bind_param() fails, it crashes.

glibc reports: double free or corruption (fasttop)

ASAN reports: heap-use-after-free

  #1 0x55ad770cda23 in mysql_bind_param libmysql/libmysql.cc:2477:9

freed by thread T0 here:

  #1 0x55ad770cda23 in mysql_bind_param libmysql/libmysql.cc:2477:9

The wrong counter-variable is used which results in the same index being
freed again and again.

Change
------

- use the right index-variable when freeing ->names if
  mysql_bind_param() fails.

Change-Id: I580267d5913c55b00151409c25d90f2ffe1b4119

Compilation warning/error:
storage/ndb/test/src/UtilTransactions.cpp:1544: error: variable 'eof'
may be uninitialized when used here [-Werror,-Wconditional-
uninitialized]

Manual code inspection also shows there are problems with
retry logic as well as potentially releasing resources.

Fix by refactoring ´verifyTableReplicasWithSource` into functions where
the retry, scan and compare logic are separated. This makes it possible
to make sure that the "eof" variable is always initialized while
fetching the next row during scan.

Change-Id: I111e0a6613622aa2279a5ec5845e48e8ca0e115f