Silence the Valgrind warnings on instrumented builds (-DWITH_VALGRIND).

assert_block_ahi_empty_on_init(): A variant of
assert_block_ahi_empty() that declares n_pointers initialized and then
asserts that n_pointers==0.

In Valgrind-instrumented builds, InnoDB declares allocated memory
uninitialized.

btr_search_drop_page_hash_index(): Do not return before ensuring
that block->index=NULL, even if !btr_search_enabled. We would
typically still skip acquiring the AHI latch when the AHI is
disabled, because block->index would already be NULL. Only if the AHI
is in the process of being disabled, we would wait for the AHI latch
and then notice that block->index=NULL and return.

The above bug was a regression caused in MySQL 5.7.9 by the fix of
Bug#21407023: DISABLING AHI SHOULD AVOID TAKING AHI LATCH

The rest of this patch improves diagnostics by adding assertions.

assert_block_ahi_valid(): A debug predicate for checking that
block->n_pointers!=0 implies block->index!=NULL.

assert_block_ahi_empty(): A debug predicate for checking that
block->n_pointers==0.

buf_block_init(): Instead of assigning block->n_pointers=0,
assert_block_ahi_empty(block).

buf_pool_clear_hash_index(): Clarify comments, and assign
block->n_pointers=0 before assigning block->index=NULL.
The wrong ordering could make block->n_pointers appear incorrect in
debug assertions. This bug was introduced in MySQL 5.1.52 by
Bug#13006367 62487: INNODB TAKES 3 MINUTES TO CLEAN UP THE
ADAPTIVE HASH INDEX AT SHUTDOWN

i_s_innodb_buffer_page_get_info(): Add a comment that
the IS_HASHED column in the INFORMATION_SCHEMA views
INNODB_BUFFER_POOL_PAGE and INNODB_BUFFER_PAGE_LRU may
show false positives (there may be no pointers after all.)

ha_insert_for_fold_func(), ha_delete_hash_node(),
ha_search_and_update_if_found_func(): Use atomics for
updating buf_block_t::n_pointers. While buf_block_t::index is
always protected by btr_search_x_lock(index), in
ha_insert_for_fold_func() the n_pointers-- may belong to
another dict_index_t whose btr_search_latches[] we are not holding.

RB: 13879
Reviewed-by: Jimmy Yang <jimmy.yang@oracle.com>

Analysis:
========
A foreign key constraint cannot reference a secondary index defined
on a generated virtual column. While adding new index/drop existing
column, server internally drops the internal foreign key index and
it leads to choose the virtual secondary index as foreign key index.
But innodb doesn't allow foreign key constraint reference to
secondary virtual index.

Fix:
===
Allow foreign key constraint refer to secondary index defined on
a generated virutal column.

Reviewed-by: Jimmy Yang<jimmy.yang@oracle.com>
RB: 13586

This bug was introduced in MySQL 5.6.8 with WL#6255.
When an error occurs while rebuilding a table that only has a
hidden GEN_CLUST_INDEX inside InnoDB, ha_alter_info->key_info_buffer
would be invalid and should not be dereferenced.

get_error_key_name(): Get the name of an erroneous key.
Avoid dereferencing ha_alter_info->key_info_buffer when no keys
exist in the SQL layer.

ha_innobase::inplace_alter_table(),
ha_innobase::commit_try_rebuild(): Invoke get_error_key_name()
for reporting ER_INNODB_ONLINE_LOG_TOO_BIG or ER_INDEX_CORRUPT.

RB: 13834
Reviewed-by: Jimmy Yang <jimmy.yang@oracle.com>

When the rows produced on the current iteration are sent to the
temporary table T of the UNION type created for CTE the rows
that were not there simultaneously are sent to the temporary
table D that contains rows for the next iteration. The test
whether a row was in T checks the return code of writing into T.
If just a HEAP table is used for T then the return code is
HA_ERR_FOUND_DUPP_KEY, but if an ARIA table is used for T then
the return code is HA_ERR_FOUND_DUPP_UNIQUE.
The implementation of select_union_recursive::send_data()
erroneously checked only for the first return code. So if an Aria
table was used for T then all rows produced by the current iteration
went to D and and in most cases D grew with each iteration.
Whether T has reached stabilization is detected by
checking whether D is empty. So as a result, the iterations were
never stopped unless a limit for them was set.

Fixed by checking for both HA_ERR_FOUND_DUPP_KEY and
HA_ERR_FOUND_DUPP_UNIQUE as return codes returned by
the function writing a row into the temporary table T.

When WSREP(thd) is not true we will use my_error(...) to print error. This
will set thd->is_error() to true and we wont be getting generic error.
Signed-off-by: Sachin Setiya <sachin.setiya@mariadb.com>

In the current code temporary tables we identified and opened before
other tables. CTE tables are identified in the same procedure as
regular tables. When a temporary table and a CTE table have the same
name T any reference to T that is in the scope of the CTE declaration
must be associated with this CTE. Yet it was not done properly.
When a reference to T was found in the scope of the declaration
of CTE T a pointer to this CTE was set in the reference. No check
that the reference had been already associated with a temporary table
was done. As a result, if the temporary table T  had been created then
the reference to T was considered simultaneously as reference to the CTE
named T and as a reference to the temporary table named T. This
confused the code that were executed later and caused a crash of
the server.
Now when a table reference is associated with a CTE any previous
association with a temporary table is dropped.

This problem could be easily avoided if the temporary tables were
not identified prematurely.
as reference to CTE named T and

Signed-off-by: Daniel Black <daniel.black@au.ibm.com>

When we build server out of git directory , we did not copy
wsrep_sst_common into build-dir. This causes galera/wsrep tests
to fail

Condition pushdown into derived tables / views with side effects
is not allowed.

Here's  what started happening after the patch that fixed
the bug mdev-10454 with query reported for the bug
SELECT * FROM t t1 right JOIN t t2 ON (t2.pk = t1.pk)
  WHERE (t2.i, t2.pk) NOT IN ( SELECT t3.i, t3.i FROM t t3, t t4 )
        AND t1.c = 'foo';
The patch added an implementation of propagate_equal_fields() for
the class Item_row and thus opened the possibility of equal fields
substitutions.
At the prepare stage after setup_conds() called for WHERE condition
had completed the flag of maybe_null of the Item_row object created
for  (t2.i, t2.pk) was set to false, because the maybe_null flags of
both elements were set to false. However the flag of maybe_null for
t1.pk from the ON condition were set to true, because t1 was an inner
table of an outer join.
At the optimization stage the outer join was converted to inner join,
but the maybe_null flags were not corrected and remained the same.
So after the substitution t2.pk/t1.pk. the maybe_null flag for the
row remained false while the maybe_flag for the second element of
the row was true. As a result, when the in-to_exists transformation
was performed for the NOT IN predicate the guards variables were
not created for the elements of the row, but a guard object for
the second element was created. The object were not valid because
it referred to NULL as a guard variable. This ultimately caused
a crash when the expression with the guard was evaluated at the
execution stage.

The patch made sure that the guard objects are not created without
guard variables.

Yet it does not resolve the problem of inconsistent maybe_null flags.
and it might be that the problem will pop op in other pieces of code.
The resolution of this problem is not easy, but the problem should
be resolved in future versions.

PROBLEM

By design stats estimation always reading uncommitted data. In this scenario
an uncommitted transaction has deleted all rows in the table. In Innodb
uncommitted delete records are marked as delete but not actually removed
from Btree until the transaction has committed or a read view for the rows
is present.While calculating persistent stats we were ignoring the delete
marked records,since all the records are delete marked we were estimating
the number of rows present in the table as zero which leads to bad plans
in other transaction operating on the table.

Fix

Introduced a system variable called innodb_stats_include_delete_marked
which when enabled includes delete marked records for stat
calculations .

Apparently, WL#8149 QA did not cover the code changes made to
online table rebuild (which was introduced in MySQL 5.6.8 by WL#6255)
for ROW_FORMAT=REDUNDANT tables.

row_log_table_low_redundant(): Log the new values of indexed virtual
columns (ventry) only once.

row_log_table_low(): Assert that if o_ventry is specified, the
logged operation must not be ROW_T_INSERT, and ventry must be specified
as well.

row_log_table_low(): When computing the size of old_pk, pass v_entry=NULL to
rec_get_converted_size_temp(), to be consistent with the subsequent call
to rec_convert_dtuple_to_temp() for logging old_pk. Assert that
old_pk never contains information on virtual columns, thus proving that this
change is a no-op.

RB: 13822
Reviewed-by: Jimmy Yang <jimmy.yang@oracle.com>

Problem:
=======
Autoincrement value gives duplicate values because of the following reasons.

(1) In InnoDB handler function, current autoincrement value is not changed
based on newly set auto_increment_increment or auto_increment_offset variable.

(2) Handler function does the rounding logic and changes the current
autoincrement value and InnoDB doesn't aware of the change in current
autoincrement value.

Solution:
========
Fix the problem(1), InnoDB always respect the auto_increment_increment
and auto_increment_offset value in case of current autoincrement value.
By fixing the problem (2), handler layer won't change any current
autoincrement value.
Reviewed-by: Jimmy Yang <jimmy.yang@oracle.com>
RB: 13748

When a table has no PRIMARY KEY, but there is a UNIQUE INDEX
defined on NOT NULL columns that are not column prefixes,
that unique index must be treated as the primary key.

This property was being violated by InnoDB when a column was changed
to NOT NULL, such that a UNIQUE INDEX on that column became eligible
to being treated as a primary key.

innobase_create_key_defs(): Instead of checking each ADD [UNIQUE] INDEX
request, check if a GEN_CLUST_INDEX can be replaced with any unique index
in the altered_table definition. So, we can have new_primary even
if n_add==0.

prepare_inplace_alter_table_dict(): When the table is not being rebuilt,
assert that TABLE_SHARE::primary_key is not changing.

RB: 13595
Reviewed-by: Kevin Lewis <kevin.lewis@oracle.com>

We don't free some memory used by fts optimize thread in release build.

Patch approved by Allen.Lai over IM.

Problem:
=======
High priority transaction can't able to kill the blocking transaction
when foreign keys are involved. trx_kill_blocking() missing while checking
the foreign key constraint.

Fix:
===
Add trx_kill_blocking() while checking for the foreign key constraint.
Reviewed-by: Debarun Banerjee <debarun.banerjee@oracle.com>
RB: 13579

Analysis: In row_log_table_delete(), extern size could be greater
than 2 bytes int if there are enough index on blob columns.

Solution: Use 4 bytes int other than 2 bytes for extern size.
Reviewed-by: Marko Mäkelä <marko.makela@oracle.com>
RB: 13573

Backport WL#9383 INNODB: ADD AN OPTION TO TURN OFF/ON DEADLOCK CHECKER
(rb#12873) to 5.7.

BUG#23742339 FAILING ASSERTION: SYM_NODE->TABLE != NULL

Analysis: When we access fts aux tables in information schema,the
fts aux tables are dropped by DROP DATABASE in another session.

Solution: Drop parent table if it's a fts aux table, and drop
table will drop fts aux tables together.
Reviewed-by: Jimmy Yang <jimmy.yang@oracle.com>
RB: 13264

Analysis:
the old table is dropped, just after it's added into drop list,
and new table with the same name is created, then we try to drop
the new table in background.

Solution:
Don't drop a table in background if table->to_be_dropped is false.
Reviewed-by: Jimmy Yang <jimmy.yang@oracle.com>
RB: 13414

Analysis:
When we access fts_internal_tbl_name in i_s_fts_config_fill (),
it can be set to NULL by another session.

Solution:
Define fts_internal_tbl_name2 for global variable innodb_ft_aux_table,
if it's NULL, set fts_internal_tbl_name to "default".
Reviewed-by: Jimmy Yang <jimmy.yang@oracle.com>
RB: 13401

Problem:
=======
Inplace alter algorithm determines the table to be rebuild if the table
undergoes row format change, key block size if handler flag contains only
change table create option. If alter with inplace ignore flag operations and change table create options then it leads to table rebuild operation.

Solution:
========
During the check for rebuild, ignore the inplace ignore flag and check for
table create options.
Reviewed-by: Jimmy Yang <jimmy.yang@oracle.com>
Reviewed-by: Marko Makela <marko.makela@oracle.com>
RB: 13172

When the WHERE condition is always FALSE then JOIN::join_tab is NULL.

This patch fixed some problems that occurred with subqueries that
contained directly or indirectly recursive references to recursive CTEs.

1. A [NOT] IN predicate with a constant left operand and a non-correlated
subquery as the right operand used in the specification of a recursive CTE
was considered as a constant predicate and was evaluated only once.
Now such a predicate is re-evaluated after every iteration of the process
that produces the records of the recursive CTE.
2. The Exists-To-IN transformation could be applied to [NOT] IN predicates
with recursive references. This opened a possibility of materialization
for the subqueries used as right operands. Yet, materialization
is prohibited for the subqueries if they contain a recursive reference.
Now the Exists-To-IN transformation cannot be applied for subquery
predicates with recursive references.

The function st_select_lex::check_subqueries_with_recursive_references()
is called now only for the first execution of the SELECT.

fil_compress_page(): Check for space==NULL.

buf_flush_write_block_low(): Acquire the tablespace reference once,
and pass it to lower-level functions. This is only a start; further
calls may be removed.

fil_decompress_page(): Remove unsafe use of fil_space_get_by_id().

fil_crypt_thread(): Do invoke fil_crypt_complete_rotate_space()
when the tablespace is about to be dropped. Also, remove a redundant
check whether rotate_thread_t::space is NULL. It can only become
NULL when fil_crypt_find_space_to_rotate() returns false, and in
that case we would already have terminated the loop.

fil_crypt_find_page_to_rotate(): Remove a redundant check for
space->crypt_data == NULL. Once encryption metadata has been
created for a tablespace, it cannot be removed without dropping
the entire tablespace.

On 64-bit Windows, sizeof(ulint)!=sizeof(ulong).

Alias the InnoDB ulint and lint data types to size_t and ssize_t,
which are the standard names for the machine-word-width data types.

Correspondingly, define ULINTPF as "%zu" and introduce ULINTPFx as "%zx".
In this way, better compiler warnings for type mismatch are possible.

Furthermore, use PRIu64 for that 64-bit format, and define
the feature macro __STDC_FORMAT_MACROS to enable it on Red Hat systems.

Fix some errors in error messages, and replace some error messages
with assertions.
Most notably, an IMPORT TABLESPACE error message in InnoDB was
displaying the number of columns instead of the mismatching flags.

Define UNIV_WORD_SIZE as a simple alias to SIZEOF_SIZE_T.
In MariaDB 10.0 and 10.1, it was incorrectly defined as 4 on
64-bit Windows.

MONITOR_OS_PENDING_READS, MONITOR_OS_PENDING_WRITES: Enable by default.

os_n_pending_reads, os_n_pending_writes: Remove.
Use the monitor counters instead.

Use older version of the SDK generally, because the newer ones break
with older cmake.

On Macs, use newer version, to fix mac specific the build error.

Update AWS SDK version from 1.0.8 to 1.0.100

Commit b64910ce (MDEV-12453)
enabled AWS_SDK to build correctly on buildbot.

Travis still had build faults like below despite many common elements
between the builds;

/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../include/c++/v1/cstring:79:9: error: no member named 'strcoll' in the global namespace; did you mean 'strtoll'?
[ 24%] Building CXX object storage/rocksdb/CMakeFiles/rocksdblib.dir/rocksdb/db/internal_stats.cc.o
using ::strcoll;
      ~~^
/usr/include/stdlib.h:169:3: note: 'strtoll' declared here
         strtoll(const char *__str, char **__endptr, int __base);
         ^

When MDEV-6076 repurposed the field PAGE_MAX_TRX_ID, it was assumed
that the field always was 0 in the clustered index of old data files.
This was not the case in IMPORT TABLESPACE (introduced in MySQL 5.6
and MariaDB 10.0), which is writing the transaction ID to all index
pages, including clustered index pages.

This means that on a data file that was at some point of its life
IMPORTed to an InnoDB instance, MariaDB 10.2.4 or later could interpret
the transaction ID as a persistent AUTO_INCREMENT value.

This also means that future changes that repurpose PAGE_MAX_TRX_ID
in the clustered index may cause trouble with files that were imported
at some point of their life.

There is a separate minor issue that InnoDB is writing PAGE_MAX_TRX_ID
to every secondary index page, even though it is only needed on leaf
pages. From now on we will write PAGE_MAX_TRX_ID as 0 to non-leaf pages,
just to be able to keep stricter debug assertions.

btr_root_raise_and_insert(): Reset the PAGE_MAX_TRX_ID field on non-root
pages of the clustered index, and on the no-longer-leaf root page of
secondary indexes.

AbstractCallback::is_root_page(): Remove. Use page_is_root() instead.

PageConverter::update_index_page(): Reset the PAGE_MAX_TRX_ID to 0
on other pages than the clustered index root page or secondary index
leaf pages.

Fixed handling of default values with cached temporal functions so that the
CREATE TABLE statement now succeeds.
Fixed virtual column session cleanup.
Fixed the error message.
Added quoting of date/time values in cases when this was omitted.
Added a test case in default.test.
Updated test result files.