statistics calculation

 Analysis: When --replace or --insert-ignore is not given, dumping of
mysql.innodb_index_stats and mysql.innodb_table_stats will result into race
condition.
Fix: Check if these options are present with --system=stats (because dumping
under --system=stats is safe). Otherwise, dump only structure, ignoring data
because innodb will recalculate data anyway.

recv_recovery_from_checkpoint_start(): Clear the recv_no_ibuf_operations
flag at the same time when we enabled writes to the log.

The failure to clear the flag might have caused some missed
change buffer merges, at least to the secondary index of SYS_TABLES
that were accessed by trx_resurrect_table_locks() while the last
recovery batch was in progress.

Thanks to Thirunarayanan Balathandayuthapani for suggesting this fix.

fil_rename_tablespace(): Do not write a redundant MLOG_FILE_RENAME2
record.

The recovery bug will be fixed later. The problem is that we are
invoking fil_op_replay_rename() too often, while we should skip
any 'intermediate' names of a tablespace and only apply the
very last rename for each tablespace identifier, and only if
the tablespace name is not already correct.

It seems that memory is not freed when updated value is NULL or
when wsrep is not initialized before shutdown.

Update mysqldump man page for --system={stats,timezones}

statistics calculation

mysqldump --system=stats and --system=timezones by default used
ordinary INSERT statements populate EITS, innodb stats, and timezone tables.

As these all have primary keys it could result in conflict.

The behavior desired with --system= is to replace the tables.
As such we assume --replace for the purposes of stats and timezone tables
there if --insert-ignore isn't specified.

mutex order violation here.
when wsrep bf thread kills a conflicting trx, the stack is

  wsrep_thd_LOCK()
  wsrep_kill_victim()
  lock_rec_other_has_conflicting()
  lock_clust_rec_read_check_and_lock()
  row_search_mvcc()
  ha_innobase::index_read()
  ha_innobase::rnd_pos()
  handler::ha_rnd_pos()
  handler::rnd_pos_by_record()
  handler::ha_rnd_pos_by_record()
  Rows_log_event::find_row()
  Update_rows_log_event::do_exec_row()
  Rows_log_event::do_apply_event()
  Log_event::apply_event()
  wsrep_apply_events()

and mutexes are taken in the order

  lock_sys->mutex -> victim_trx->mutex -> victim_thread->LOCK_thd_data

When a normal KILL statement is executed, the stack is

  innobase_kill_query()
  kill_handlerton()
  plugin_foreach_with_mask()
  ha_kill_query()
  THD::awake()
  kill_one_thread()

and mutexes are

  victim_thread->LOCK_thd_data -> lock_sys->mutex -> victim_trx->mutex

To fix the mutex order violation we kill the victim thd asynchronously,
from the manager thread

and void wsrep_innobase_kill_one_trx()

as their return values are never used.
Also remove redundant cast and checks that are always true

* provide an argument to the callback
* don't ignore a callback request if it's already present in the queue
* initialize mutex/cond/in_use flag before starting the thread,
  in case the first callback queueing request arrives before
  handle_manager had time to initialize
* set/check abort_manager under a mutex, otherwise handle_manager
  thread might destroy LOCK_manager before stop_handle_manager
  released it
* signal COND on queueing a callback, stop cond_wait on callback request
* always start the thread, even if flush_time is 0
* but keep the old behavior in embedded (no replication, no galera)
* style cleanups (e.g. remove volatile for a variable protected by a mutex)

MDEV-23201 correctly reordered columns in mysql.user table when upgrading
from MySQL-5.7

Here we also correctly reorder columns in mysql.user table from an invalid order
caused by an upgrade from MySQL-5.7 to MariaDB-before-MDEV-23201.

Update
* galera/disabled.def
* galera_3_nodes/disabled.def

Clarified working contributed by Vicentiu

Sample log error message generated:

2021-01-21  2:33:24 139912137520896 [Note] Slave SQL thread exiting, replication stopped in log 'master-bin.000001' at position 369
33:24 139912137520896 [Note] master was 127.0.0.1:16400
2021-01-21  2:33:24 139912137828096 [Note] Slave I/O thread exiting, read up to log 'master-bin.000001', position 369
2021-01-21  2:33:24 139912137828096 [Note] master was 127.0.0.1:16400

Based on work by Hartmut Holzgraefe.

Reviewer: knielsen@knielsen-hq.org, Andrei, Sachin

Problem:
========
Auto purge of relaylogs stops when relay-log-file is
'slave-relay-log.999999' and slave_parallel_threads is enabled.

Analysis:
=========
The problem is that in Relay_log_info::inc_group_relay_log_pos() function,
when two log names are compared via strcmp() function, it gives correct
result, when log name sequence numbers are of same digits(6 digits), But
when the number goes to 7 digits, a 999999 compares greater than
1000000, which is wrong, hence the bug.

Fix:
====
Extract the numeric extension part of the file name, convert it into
unsigned long and compare.

Thanks to David Zhao for the contribution.

Per cppcheck, va_start is called twice which it is.

Remove the second instance.

           through 2nd execution of SP

This bug caused a server crash on the second call of any stored procedure
that contained an UPDATE statement over a multi-table view reporting an
error message at the prepare stage.
On the first call of the stored procedure after reporting an error at
the preparation stage of the UPDATE statement finished without calling
the function SELECT_LEX::save_prep_leaf_tables() for the SELECT used as
the definition of the view. This left the SELECT_LEX structure used by
the UPDATE statement in an inconsistent state for second call of the stored
procedure.

Approved by Oleksandr Byelkin <sanja@mariadb.com>

There are only two InnoDB changes between mysql-5.7.32 and mysql-5.7.33:

mysql/mysql-server@95dc4f5f08e479262234498490de2d64cfe9ebc9
duplicates commit 8e8e65ed (MDEV-10829).

mysql/mysql-server@26e849762f1b2f79434bd3e6e611beccec39efc9
could be an attempt to fix something that was fixed in
commit dc58987e (MDEV-22765).

innodb_io_capacity_update(): When the requested innodb_io_capacity
exceeds innodb_io_capacity_max and is more than half the maximum,
do not double it for computing innodb_io_capacity_max.

This integer arithmetics overflow was introduced in
commit 0f322994 (MDEV-7035).

No test case is added, because sizeof(ulong) varies between platforms.

MDEV-19723 Assertion `je->state == JST_KEY' failed while SELECT ST_GEOMFROMGEOJSON() and Assertion `!mysql_bin_log.is_open() || thd.is_current_stmt_binlog_format_row()'

The invalid GeoJSON case wasn't handled here.

Ignore table->fts freed previously by create_table_info_t::create_table().

    Database name mismatch happens while opening the table for
virtual column computation. Because table_name_parse() returns
the length of database and table name before converting the
filename to table name. This issue is caused by
commit 8b0d4cff (MDEV-15855).
Fix should be that table_name_parse() should return the length of
database and table name after converting the filename to table name.

Reviewed-by: Marko Mäkelä

When online alter rollbacks due to MDL time out, it doesn't mark the
index online status as ONLINE_INDEX_ABORTED. Concurrent update fails
to update the secondary index while building the entry.
InnoDB should check the online status of the secondary index before
building the secondary index entry.

Reviewed-by: Marko Mäkelä

m_status == DA_OK_BULK' failed in Diagnostics_area::message()

Analysis: Assertion failure happens because we reach the maximum limit to
examine rows.
Fix: Return the error state.

Some DML operations on tables having unique secondary keys cause scanning
in the secondary index, for instance to find potential unique key violations
in the seconday index. This scanning may involve GAP locking in the index.
As this locking happens also when applying replication events in high priority
applier threads, there is a probabality for lock conflicts between two wsrep
high priority threads.

This PR avoids lock conflicts of high priority wsrep threads, which do
secondary index scanning e.g. for duplicate key detection.

The actual fix is the patch in sql_class.cc:thd_need_ordering_with(), where
we allow relaxed GAP locking protocol between wsrep high priority threads.
wsrep high priority threads (replication appliers, replayers and TOI processors)
are ordered by the replication provider, and they will not need serializability
support gained by secondary index GAP locks.

PR contains also a mtr test, which exercises a scenario where two replication
applier threads have a false positive conflict in GAP of unique secondary index.
The conflicting local committing transaction has to replay, and the test verifies
also that the replaying phase will not conflict with the latter repllication applier.
Commit also contains new test scenario for galera.galera_UK_conflict.test,
where replayer starts applying after a slave applier thread, with later seqno,
has advanced to commit phase. The applier and replayer have false positive GAP
lock conflict on secondary unique index, and replayer should ignore this.
This test scenario caused crash with earlier version in this PR, and to fix this,
the secondary index uniquenes checking has been relaxed even further.

Now innodb trx_t structure has new member: bool wsrep_UK_scan, which is set to
true, when high priority thread is performing unique secondary index scanning.
The member trx_t::wsrep_UK_scan is defined inside WITH_WSREP directive, to make
it possible to prepare a MariaDB build where this additional trx_t member is
not present and is not used in the code base. trx->wsrep_UK_scan is set to true
only for the duration of function call for: lock_rec_lock() trx->wsrep_UK_scan
is used only in lock_rec_has_to_wait() function to relax the need to wait if
wsrep_UK_scan is set and conflicting transaction is also high priority.
Reviewed-by: Jan Lindström <jan.lindstrom@mariadb.com>

MDEV-24432 : galera.galera_fk_cascade_delete_debug MTR failed: query 'reap' failed: 1205: Lock wait timeout exceeded

Add wait_conditions to verify correct database state before
next operation.

Use debug_sync to force FTWRL to pause in correct state.

MDEV-12908 binlog_encryption.binlog_xa_recover, binlog.binlog_xa_recover failed in bb with extra checkpoint

The parameter innodb_idle_flush_pct that was introduced in
MariaDB Server 10.1.2 by MDEV-6932 has no effect ever since
the InnoDB changes from MySQL 5.7.9 were applied in
commit 2e814d47.

Let us declare the parameter as deprecated and having no effect.

It is unclear for how long the result has been broken, because
./mtr --big-test is not run regularly.

Added sending of metadata in response to preparing request for
the commands SQLCOM_SHOW_BINLOG_EVENTS, SQLCOM_SHOW_RELAYLOG_EVENTS

Introduced val_time_packed and val_datetime_packed  functions for Item_direct_ref
to make sure to get the value from the item it is referring to.

The issue for incorrect result was that the item was getting its value
from the temporary table rather than from the view.

For EITS collection min and max fields are allocated for each column
that is set in the read_set bitmap of a table. This allocation of min and max
fields happens inside alloc_statistics_for_table.

For a partitioned table ha_rnd_init is called inside the function
collect_statistics_for_table which sets the read_set bitmap for the columns
inside the partition expression. This happens only when there is a write lock
on the partitioned table.
But the allocation happens before this, so min and max fields are not allocated
for the columns involved in the partition expression.
This resulted in a crash, as the EITS statistics were collected but there was
no min and max field to store the value to.

The fix would be to call ha_rnd_init inside the function alloc_statistics_for_table
that would make sure that min and max fields are allocated for the columns
involved in the partition expression.