An aggregate query on a BIT field returns a value which is formatted
as a bit string, but has the BINARY flag. This is because the
BINARY flag is automatically added to all result items which have a
binary charset. This should not be done for BIT results since they
are always formatted as bit strings.

This is fixed by adding a check to skip setting the BINARY flag for
BIT results.

Change-Id: If73f511cb27a4e63436bf7e48305bf7b7d197859

              optimizer is ignoring the STRAIGHT_JOIN HINT

Contrary to what the bug title says, hypergraph is not ignoring the
straight_join hint. Since straight joins are associative,
it still produces a valid plan as per the directives.
However, it does not pick the ideal plan. When hints are
not present, it does indeed pick a much better plan.
But, for heatwave this is a problem because of the number
of partial plans that are generated.
The problem when straight join hints are given is that only a
few combinations are possible. And with these limited
combinations and the order in which the join conditions
get pushed down to each of these joins, makes the combination
for the best join not possible for this particular case.
E.g.
For a query of this form (SJ - straight_join):
t1 SJ t2 SJ t3 SJ t4 with 4 join conditions:
t3->t4 has a non equality join condition
t2->t4 has an equijoin condition
t1->t3 has an equijoin condition
t1->t2 has an equijoin condition
Call to optimize_cond() places all the equalities at the end of
the final where/join condition. So the in-equality is placed first
in this case.
When optimizer is trying to push the conditions down to joins
initially, it sees the non equality join condition first and
pushes it down for a join between t3->t4 (2 large tables in case
of the non-performant TPC-DS query).
Then it sees the (t2->t4) equi-join condition. It concludes that
it's better to place this condition for the join between (t2)->(t3t4)
because of 2 reasons. The first one being that the used tables
for the join t3->t4 has t3,t4 because of the already present join
condition and the second one being that t3 needs to be seen
before t4 (because of stright_join).
For the next join condition between (t1->t3), it places it on the
join between (t1,(t2t3t4)) and the same with the last condition.
The same reasoning persists when creating edges resulting
in the join between (t3,t4) which are large tables in the actual
use case.
However, if the optimizer were to see the equalities first,
it would place the join condition in such a way that the following
join was possible ((t1 HJ t2) HJ t3) HJ t4) which is still valid as
per the hints but a much better join.
The same problem does not happen when the hints are not used
because many other joins are possible and therefore it could create
more edges.
It could be argued that optimizer should have been able to figure
out the valid edge between t1->t3. However its easier to fix this
particular case by just sorting conditions in a way that the
equalities are seen earlier. So we take the easier route for now.
And putting equijoin conditions ahead of others does no harm.
Also that at this point, the predicates are not ordered in any way
except for optimize_cond() doing a bit of sorting for the multiple
equalties. So in this patch, we put all the equijoin conditions
at the begining followed by other types of conditions. We also make
sure that the sorting that was done while creating hash join
conditions is taken care here itself i.e putting all the conditions
with subqueries and the expensive conditions at the end of the list.
This results in a few failing unit tests. The necessary changes have
been done for those tests.

Change-Id: I6880ffbe6b3fa177a3fbb7612aed3f46eb6440d7

Upgrade the bundled googletest/googlemock source to the most recent version,
which is currently 1.12.0

The workaround for -Werror=maybe-uninitialized for googlemock matchers
can be removed, the bug has been fixed upstream.

git add extra/googletest/googletest-release-1.12.0/googlemock/
git add extra/googletest/googletest-release-1.12.0/googletest/
git rm -rf extra/googletest/googletest-release-1.12.0/googlemock/test
git rm -rf extra/googletest/googletest-release-1.12.0/googletest/test

git rm -rf extra/googletest/googletest-release-1.11.0

Visual Studio fails to compile any usage of testing::WithArgs<0, ...>
gmock-actions.h(1432,59):error C2039: 'type' is not a member of 'std'

The workaround is to add extra arguments to:
  Mock_dd_field_varstring::fake_store()
  Mock_dd_field_varstring::fake_val_str()
so that we can remove WithArgs<0> and WithArgs<1>.

Similarly for
  mock_gcs_xcom_state_exchange_interface::free_xcom_member_state()
  mock_gcs_xcom_state_exchange_interface::free_members_joined()
  suspect_is_me()
  suspect_is_2()
  copy_message_content()

The macro GTEST_DISALLOW_COPY_AND_ASSIGN_ is no longer provided by
googletest headers. The fix is to expand the old definition of the
macro at all places it was used.

Some source file need COMPILE_FLAGS "/bigobj" with Visual Studio.

Change-Id: Id078f63163e672ac6949901c664a712a00ec4d72
(cherry picked from commit 69ef1a1621b85be54dbdb32a26acaee099f40f6c)

Additional patch:
remove obsolete comments for GTEST_DISALLOW_COPY_AND_ASSIGN_

Change-Id: Ice549fff87b3a2a87d961d3e46863e0e38b983b4
(cherry picked from commit 32fdf87cb3380582a5d8faa917101c8d8b7999c5)

…44 to mysql-trunk

Backport from mysql-trunk.

All queries where there are UNION DISTINCT followed by UNION ALL were
not offloaded on mysql-trunk. Queries are of the form
1. (SELECT ...) UNION DISTINCT (SELECT ...) UNION ALL (SELECT ...)
2. CREATE VIEW V1 AS (SELECT ...) UNION
         DISTINCT (SELECT ...) UNION ALL (SELECT ...)

Problem: In the queries with UNION ALL, STREAM Access Path did not find
the correct JOIN Pointer/ Query Block, and hence did not find the correct
compilation context, which lead to failing assertions.

Solution:
Since Union(all) is a SET operation and query_terms are used in
MYSQL to represent SET Operations, and it is available on mysql-trunk,
we use query_terms to resolve query_blocks for union children.
This solution introduces query_term in the TranslateState to facilitate
this solution and the solution to BUG#34162247. Traversing query_term
tree additionally helps improves the following:
1. Readibility: Less corner cases to check for
     when accessing query blocks in presence of Set Operations.
2. Maintanability: Fewer single use code paths necessary to obtain
     correct query blocks. This will be fully accomplished after
     closing BUG#34323639.
3. Speed: Traversing query term tree is faster than traversing
     AP tree in some cases, since AP trees can be nested very
     deep, whereas, query term trees nest only as much as
     maximum depth of set operations.
Conceptually, this solution also helps to resolve the following
challenges which arise in translating the access path with
query terms:
1. Find appropriate query term,
   when multiple query terms are nested in 1 Query Expression and
2. Find appropriate Union Child's Join pointer
   when we have nested query terms.

Change-Id: I69f1028a71eff8fd45ff55f73a0ee51273476e75

…Query_terms Printing QueryTerm AST in APTree [1/2]

Backport from mysql-trunk.

* Print Query-term tree along with AccessPath.

* Replaces Query-block with appropriate Query-terms on need basis.

* Introduces parent and child relation between Query-term nodes to represent Query-term AST.

* Example:
  Query: CREATE VIEW v AS ((SELECT a1, a2 FROM A UNION DISTINCT SELECT b1, b2 FROM B ORDER BY a1 ASC LIMIT 10) ORDER BY a1,a2 DESC LIMIT 7);
  APTree:
  -> (qt_query_block) ptr=0x7f09f4e16e98 parent=(nil) num_children=0 m_block=0x7f09f4e16e98 select_number=1 join=0x7f09f52c8780 master_expression_qt=0x7f09f4e16e98 MATERIALIZE (0x7f09f52da6f8) (derived table)
    -> (table_path) TABLE_SCAN (0x7f09f52da340) (on v)
    -> (qt_unary) ptr=0x7f09f52bced0 parent=(nil) num_children=1 m_block=0x7f09f52bdb60 m_block_id=5 m_block_join=0x7f09f52ca2e8 master_expression_qt=0x7f09f52bced0 limit=7 offset=0 LIMIT_OFFSET (0x7f09f52cb2d8) (limit=7, offset=0)
        -> SORT (0x7f09f52cb1f0)
            -> MATERIALIZE (0x7f09f52d9b18) (temporary table)
                -> (table_path) TABLE_SCAN (0x7f09f52cb160) (on <result temporary>)
                -> (qt_union) ptr=0x7f09f4e14770 parent=0x7f09f52bced0 num_children=2 m_block=0x7f09f4e14bf0 m_block_id=4 m_block_join=0x7f09f52cb368 master_expression_qt=0x7f09f52bced0 limit=10 offset=0 LIMIT_OFFSET (0x7f09f52cc340) (limit=10, offset=0)
                    -> SORT (0x7f09f52cc258)
                        -> MATERIALIZE (0x7f09f52ccd48) (union)
                            -> (table_path) TABLE_SCAN (0x7f09f52cc1c8) (on <union temporary>)
                            -> (qt_query_block) ptr=0x7f09f4e2cca0 parent=0x7f09f4e14770 num_children=0 m_block=0x7f09f4e2cca0 select_number=2 join=0x7f09f52c8c08 master_expression_qt=0x7f09f52bced0 TABLE_SCAN (0x7f09f52c96e8) (on A)
                            -> (qt_query_block) ptr=0x7f09f4e12eb8 parent=0x7f09f4e14770 num_children=0 m_block=0x7f09f4e12eb8 select_number=3 join=0x7f09f52c9778 master_expression_qt=0x7f09f52bced0 TABLE_SCAN (0x7f09f52ca258) (on B)

Change-Id: I592e95f98b1c9b25b07d76a53957aa8bdbaaa7fc

Another case of missing thd error check, this time in
Json_table_column::fill_column()

Change-Id: I06adb5cd685a8076393edcba505210eb22f60e92
(cherry picked from commit 74f95358938e2448068265775c0d652ae4831f5f)

Post-push fix for broken build with gcc 13.0.0

include/hexify.h:47:42: error:
ISO C++ forbids declaration of type name with no type [-fpermissive]
einterpret_cast<const uint8_t *>

Change-Id: I5b506cd8293975f33ec8d3eebcaafa5aaa434d93
(cherry picked from commit 30de6e96d17fecb412f31e106a771a743876875d)

GROUP CONCAT - (without orderby)
===============================================
Support for Group concat aggregation in Heatwave.
-Group_concat(expr) returns a string result with the concatenated
non-NULL values from a group.

High level Design:
==================
All the input columns concatenated  with CONCAT expression. Concatenated
string is combined over rows by applying group_concat aggregation.

A separator is passed as the second argument to the group concat and
it is added while concatenating the rows. The passing of the separator
string also requires the removal of the limitation of having only one
input to the aggregate operator.

Session parameter group_concat_max_len is not supported beyond maximum
column width.

Current limitations:
-> Group-concat with roll-up queries.
-> Group-concat functions with ORDER BY is not supported.
-> Dictionary encoded columns are not supported.

Tests are added as part of func_gconcat.test
Result files are updated with new offloads to Heatwave.

All QA identified bugs are fixed

Change-Id: I77dd330dc4790aeba02e17db50347f1dbd8a3d8b

Avoid test failure due to timeouts.

Timeout limit was set to 20mins, while test seems to take at
least > 15min and oftentimes more. Patch double the timeout limits.

Change-Id: Id041966092579109560d212d4e4c014307401c30

Change-Id: I3b8ba5235b564b5da4b46450c94fdabb3f9fc544

…ential overhead.

The Parallel(lock) queue lists in DbAcc will contain all shared-locks granted
to the 'same row', when the row is read multiple times using an unique key in
the same transaction (or query). In particular this may happen when the same
row is joined with other rows in a large join query.

WITH_DEBUG compiled binaries, as well as binaries compiled for ERROR_INSERT,
 will 'validate' the lock queues in order to check the consistency,
which involves traversing the entire lock lists.
This adds an exponential overhead to the code, which could
ends up totally stalling the data nodes. 'Overslept' warnings, upto
node disconnects and arbitrator shuttig down nodes may be seen.

Patch adds an upper limit (of 42) of how many queue items we validate
before giving it a 'pass'.

We usually seems to insert, remove and *validate* the lists starting
either from the 'lockOwner' (The head) or the last item, and validate
the lists from this starting position. The list contents inbetween
the head and tail has few changes. Thus, not validate the entire
huge lists should not loose much test coverage.

Change-Id: Ib3554844ecd5c143bca995bbaa83cac2ff2653df
(cherry picked from commit a677c64bf759cd7446005db9f66aa1c1f904d06a)

…ra predicate

A condition like t1.a = t2.a + t3.a is not considered an equi-join condition
because the right side of the condition has more than one table. As a result,
this makes it to the extra predicate of the hash join. However, if the
optimizer knows that "t2.a = t3.a", the above condition could well be
transformed into t1.a = t2.a + t2.a which could be used as an equi-join
condition. This is done in the old optimizer after the join order is decided.
Hence heatwave is able to offload in case of the old optimizer and not in
Hypergraph.
Solution is to propagate such equalities into all conditions that are not
multiple equalities. This results in several unit tests failing because some
of these fields in non-equalities were getting replaced with another table's
field or the conditions were written in such a way that optimizer could now
detect that these conditions are going be always false. E.g. t1.x = t2.x
and t1.x <> t2.x would now be transformed into t1.x = t2.x and t1.x <> t1.x.
To retain the original intention of the test case, all such cases are changed
to using different field from the same table.

Change-Id: Iaf9c8b393bb563867c6ac56bc637804780648269

derived table transformation missing

Post-push fix: The calculation of disallowed tables mixed NodeMap and
table_map. The node numbers and the table numbers are often the same,
but not always, so this could cause incorrect behaviour.

Fixed it by using table_map consistently.

Change-Id: I2242c651711c793e769063ec1d1da1351c18f4f8

by Bug#33732907).

This calls back to the fact that when running in "normal mode",
we see one COM_QUERY row for each statement, whereas when running
with --ps-protocol, we see a COM_STMT_PREPARE/COM_STMT_EXECUTE
pair for each statement. This affects counters and log file/table
contents.

This test now accounts for the mode it runs in and adjusts row
counts accordingly. Since we no longer show individual log lines
in the result, we now take separate counts of lines of interest
with and lines of interest without obfuscated passwords.

Change-Id: Id267a87f4ae1df835a2974dfc405d9aeee0ca8fe

…lgorithm used

  Issue:
    Tables with INSTANT ADD/DROP columns does not support
    tablespace import without the .cfg file. The code is
    throwing an error in the release build (as expected)
    but hitting a debug assert in the debug build. The debug
    build should throw similar error instead of hitting an
    assert. The behavior should be consistent in both
    release and debug build.

  Fix:
    Raise the error at the appropriate location and handle
    the same.

Change-Id: Ifd15f8a6285183f9ecbba08a8d2b56fb62790c86

               stored functions in a connection

Description:
============
Consider a stored routine having a SELECT as a substatement
that contains multiple and/or/xor conditions. Executing the
stored routine large number of times leads to high
consumption of virtual memory eventually leading to OOM.

Analysis:
=========
Query_block is part of the AST that represents the query (a
SELECT), and it is created and populated when the statement
is prepared. However, it is not supposed to be updated after
preparation, for each execution it is supposed to be
read-only. For a regular statement, the whole AST is torn
down after preparation and execution. That's not the case
for stored routines.

The consequence is that one of its attributes
(Query_block::cond_count) which stores the number of
arguments of and/or/xor in where/having/on clause(s) in the
SELECT query, is not reset for each execution of the stored
routine. For each execution, the attribute's value gets
multiplied. This attribute's value is used to determine the
memory allocation size for Key_field structure in
update_ref_and_keys(). The memory grows linearly as the
number of calls to this function increases.

Fix:
====
Query_block::cond_count is saved during the preparation,
incremented appropriately during the optimization and
restored with the saved value during the next execution of
the stored routine.

Change-Id: I59d38d5015b03111dbe4b633d456bee809942173

               SUBQUERY - REGRESSION

Description:
Mysqlpump may not work in cases where derived tables are generated.
This is due to privileges not being set for those tables.

Fix:
Backport. Derived tables handled separately and privileges set.

Gerrit#16933

Change-Id: Ia31d479ed2d7aba47993d2e646efb84651ee784a

…filter condittion

Strings with lots of leading zeros were incorrectly marked with
overflow when converting to decimal. The value returned was the
maximum decimal value for the given precision and scale.

The fix is to replace leading zeros with a single '0'.

Also convert the macro FIX_INTG_FRAC_ERROR() to an inline function
fix_intg_frac_error().

Change-Id: I371d0ebc451c5ea77f7e8ebd9e411606f9cba01f
(cherry picked from commit 7ccf63d0636477c2b494bda5262331e467d2c563)

encountering semi join + outer join
Bug#34448530: Hypergraph Offload Issue : Filter condition not pushed down

Some queries that used to be offloaded to HeatWave with the old
optimizer, were not offloaded when using the hypergraph optimizer.
These were queries that had gone through a subquery_to_derived
transformation which ended up attaching a degenerate join condition to
a left outer join.

HeatWave refuses to offload a query if an outer join condition
references only tables on the inner side of the join. Such conditions
are safe to push down on the inner side. In the failing cases,
however, it cannot be done because pushing the condition down to a
join below will just move the problem and make the join condition of
the other join inedible to HeatWave. Pulling it up into the WHERE
clause is also not an option (for correctness). So, for now, the
condition stays in the ON clause.

To work around the issue, ProposeHashJoin() is taught to detect this
case, and manually move the degenerate join condition from the join
predicate to a filter node on top of the right child of the join. This
is only done when optimizing the query for execution in a secondary
storage engine. If HeatWave learns how to handle these join
conditions, the workaround can be removed.

Change-Id: I1f2a2569afa67e1491f265685c3370df759b605b

… should be atomically set

A new structure `btr_search_prefix_info_t` is created to contain information on prefix of the row used for AHI caching.
All AHI-related fields in `buf_block_t` are moved to a new structure `buf_block_t::ahi_t`. It is better documented, includes description of latching protocol involved. This structure is accessed using `block->ahi`.
The `btr_search_prefix_info_t` is used in multiple places as atomic, with static assertion it is lockfree.
A new structure `btr_search_sys_t::search_part_t` is created to contain all state private to each AHI partition.
`btr_cur_update_in_place()` no longer takes X-latch on AHI before doing the modification.
`btr_search_update_hash_ref()` and `btr_search_update_hash_on_insert()` check if the AHI is not disabled after acquiring the X-latch. This was a race condition that could lead to incorrect state of AHI data structures.
`btr_search_drop_page_hash_index()` is better described with inline comments now, it does not take the S-latch at the begin at all, dereference of block->ahi.index is proved to be valid in all usages. It also handles the case when AHI is being disabled in meantime in correct way, without race conditions - if it is a private block, in `buf_LRU_free_page()`, we clear the block's reference, and otherwise we do nothing.
`btr_search_set_block_not_cached()` is introduced and called in places where the block->ahi.index was set to null. Now it always checks if the value was not-null and updates the index's reference count.
`btr_search_build_page_hash_index()` and `btr_search_move_or_delete_hash_entries()` no longer take S-latch on begin of processing.
`btr_search_disable()` now returns true iff the AHI was enabled and the call changed the status to disabled. This is used in `buf_pool_resize()` to judge if it should restore the status after the resize completes. A new mutex `btr_search_enabled_mutex` is used to synchronize AHI disabling and enabling, so the enabling is not executed before all AHI clearing operations are completed.
The heaps with buffer pool's blocks backed storage that are used in AHI now are storing a free block in a pointer to atomic variable. This allows the `btr_search_sys_resize()` to not leak a block that could be allocated concurrently in `btr_search_check_free_space_in_heap()`.
A race condition between `buf_LRU_free_page()` and `btr_search_drop_page_hash_index()` is removed - the `buf_LRU_free_page()` now forces page removal from AHI even if the AHI is (being) disabled.
Fixed a big bug in ut0new infrastructure on Solaris - destructors for aligned arrays were not called.
Added `ut::unique_ptr_aligned` type to allow to name a type returned by the methods allocating aligned memory.
`btr_search_disable()` now waits for all the blocks to stop reference the indexes through AHI that may be processed by some thread executing `buf_LRU_free_page()`, before allowing the AHI to be enabled again.
`rw_lock_t`'s destructor is improved so it can be called directly.

RB#27411
Change-Id: I37bb68e0c48416ca5f578ece33c61e914277c62f

Change-Id: I3e285ea7b6f83c347257ecb791b1c87a03653c4b