During a table-rebuilding online ALTER TABLE, if
dict_index_t::remove_instant() was invoked on the source table
(because it became empty), we would inadvertently change the way
how log records are written and parsed. We must keep the online_log
format unchanged throughout the whole table-rebuilding operation.

dict_col_t::def_t: Name the type of dict_col_t::def_val.

rec_get_n_add_field_len(), rec_set_n_add_field(): Define globally,
because these will be needed in row_log_table_low().

rec_init_offsets_temp(), rec_init_offsets_comp_ordinary(): Add
the parameter def_val for explicitly passing the default values
of the instantly added columns of the source table, so that
dict_index_t::instant_field_value() will not be called during
row_log_table_apply(). This allows us to consistently parse the
online_log records, even if the source table was converted
to the canonical non-instant format during the rebuild operation.

row_log_t::non_core_fields[]: The default values of the
instantly added columns on the source table; copied
during ha_innobase::prepare_inplace_alter_table()
while the table is exclusively locked.

row_log_t::instant_field_value(): Accessor to non_core_fields[],
analogous to dict_index_t::instant_field_value().

row_log_table_low(): Add fake_extra_size bytes to the record
header if the source table was converted to the canonical format
during the operation.

row_log_allocate(): Initialize row_log_t::non_core_fields.

When the semisync slave is being stopped with STOP SLAVE just after
the master was shut down it attempts to reconnect with the master
anyway per a semisync routine.  Instead of an expected error the
io-thread segfauls in mysql_real_connect() execution at

 !mysql->options.extension->async_context

check trying to reach the extension's member while mysql->options.extension is
actually and correctly NULL.
Apparently not-NULL check for mysql->options.extension was missed and
it's deployed by the patch to fix this issue.

As a bonus it also tackles an assert
   Thread 0x7f16c72148c0 (LWP 24639) 0x00007f16c53b3bf2 in __GI___assert_fail (assertion=0x55a686117558 "global_status_var.global_memory_used == 0", file=0x55a6861171e8 "/home/andrei/MDB/WTs/10.3-clean/sql/mysqld.cc", line=2201, function=0x55a68611fa80 <mysqld_exit(int)::__PRETTY_FUNCTION__> "void mysqld_exit(int)") at assert.c:101

in a new test of the patch. The reason of the assert was insufficient cleanup
in Repl_semi_sync_slave::kill_connection() which has a branch where a MYSQL instance
was left out unfred.

Introduce the configuration option innodb_log_optimize_ddl
for controlling whether native index creation or table-rebuild
in InnoDB should keep optimizing the redo log
(and writing MLOG_INDEX_LOAD records to ensure that
concurrent backup would fail).

By default, we have innodb_log_optimize_ddl=ON, that is,
the default behaviour that was introduced in MariaDB 10.2.2
(with the merge of InnoDB from MySQL 5.7) will be unchanged.

BtrBulk::m_trx: Replaces m_trx_id. We must be able to check for
KILL QUERY even if !m_flush_observer (innodb_log_optimize_ddl=OFF).

page_cur_insert_rec_write_log(): Declare globally, so that this
can be called from PageBulk::insert().

row_merge_insert_index_tuples(): Remove the unused parameter trx_id.

row_merge_build_indexes(): Enable or disable redo logging based on
the innodb_log_optimize_ddl parameter.

PageBulk::init(), PageBulk::insert(), PageBulk::finish(): Write
redo log records if needed. For ROW_FORMAT=COMPRESSED, redo log
will be written in PageBulk::compress() unless we called
m_mtr.set_log_mode(MTR_LOG_NO_REDO).

Native ALTER TABLE is never invoked on temporary tables.

This patch fixes another problem introduced by the patch for mdev-4817.
The latter changed Item_cond::fix_fields() in such a way that it could
call the virtual method is_expensive(). With the first its call
the method saves the result in Item::is_expensive_cache. For all next
calls the method returns the result from this cache. So if the item
once was determined as expensive the method always returns true.
For subqueries it's not good, because non-optimized subqueries always
is considered as expensive.
It means that the cache should be invalidated after the call of
optimize_constant_subqueries().

	At most one transaction can be active at a time for temporary
tables. There is no need to check previous version of record for the
temporary tables.

Actually if we use "set password for " command this changes the checksum
of mysql.user table
-localhost      root            Y       Y       Y       Y       Y       Y       Y       Y     YY       Y       Y       Y       Y       Y       Y       Y       Y       Y       Y       Y     $
Y       Y       Y       Y       Y       Y       Y                                       0     00       0                       N       N               0.000000
+localhost      root            Y       Y       Y       Y       Y       Y       Y       Y     YY       Y       Y       Y       Y       Y       Y       Y       Y       Y       Y       Y     Y
Y       Y       Y       Y       Y       Y       Y                                       0     00       0       mysql_native_password           N       N               0.000000
In short we replace '' with mysql_native_password
which make checksum to be different, and hence check test case fails.

So we use UPDATE mysql.user command.

Field_varstring::sql_type() did not calculate character length correctly.
Using char_length() instead of the bad code.

The problem occurs on Ubuntu where a Spider package is installed on the system
separately from the MariaDB package.  MariaDB and Spider upgrades leave the
Spider plugin improperly installed.  Spider is present in the mysql.plugin
table but is not present in information_schema.

The problem has been corrected in Spider's installation script.  Logic has
been added to check for Spider entries in both information_schema and
mysql.plugin.  If Spider is present in mysql.plugin but is not present in
information_schema, then Spider is first removed from mysql.plugin.  The
subsequent plugin install of Spider will insert entries in both mysql.plugin
and information_schema.

Author:
  Jacob Mathew.

Reviewer:
  Kentoku Shiba.

Cherry-Picked:
  Commit 0897d81 on branch bb-10.3-MDEV-15786

The problem occurs on Ubuntu where a Spider package is installed on the system
separately from the MariaDB package.  MariaDB and Spider upgrades leave the
Spider plugin improperly installed.  Spider is present in the mysql.plugin
table but is not present in information_schema.

The problem has been corrected in Spider's installation script.  Logic has
been added to check for Spider entries in both information_schema and
mysql.plugin.  If Spider is present in mysql.plugin but is not present in
information_schema, then Spider is first removed from mysql.plugin.  The
subsequent plugin install of Spider will insert entries in both mysql.plugin
and information_schema.

Author:
  Jacob Mathew.

Reviewer:
  Kentoku Shiba.

Cherry-Picked:
  Commit 0897d81 on branch bb-10.3-MDEV-15786

This helped fix MDEV-16779.

Declare all fields of purge_vcol_info_t private, and add
accessor functions.

During a table-rebuilding operation, the function table_name_parse()
can encounter a table name that starts with #sql. Here is an example
of a failure:

CURRENT_TEST: gcol.innodb_virtual_basic
mysqltest: At line 1204: query 'alter table t drop column d ' failed:
2013: Lost connection to MySQL server during query

Let us just remove these bogus debug assertions.

If the final renaming phase during ALTER TABLE never fails, it
should not do any harm to skip the purge. If it does fail, then
we might end up 'leaking' some delete-marked records in the
indexes on virtual columns of the original table, and these
garbage records would keep consuming space until the indexes are
dropped or the table is successfully rebuilt.

This is a regression caused by the fix of MDEV-15855.

purge_vcol_info_t::set_used(): Add a missing condition.

row_purge_poss_sec(): Invoke set_used() in order to
have !is_first_fetch() when retrying.

This should affect at least rec_printer() output.

Replace pairs of rw_lock_own() calls with
calls to rw_lock_own_flagged().

These calls are only present in debug builds.

There is only one temporary tablespace. Reserving a data member in
each read-write lock object for a Boolean flag seems like a huge waste,
especially because this field was only actually used in debug builds.

LatchDebug::check_order(): Compare to fil_system.temp_space->latch.

as a separate source for data

Actually MDEV-15867 and MDEV-16192 are same, Slave adds "or replace" to create
table stmt. So create table t1 is create or replace on slave. So this bug
is not because of replication, We can get this bug on general server if we
manually add or replace to create query.

Problem:- So if we try to create table t1 (same name as of temp table t1 ) via
   CREATE or replace TABLE t AS SELECT * FROM t;
Since in this query we are creating table from select * from t1 , we call
unique_table function to see whether if source and destination table are same.
But there is one issue unique_table does not account if source table is tmp table
in this case source and destination table can be same.

Solution:- We will change find_dup_table to not to look for temp table if
CHECK_DUP_SKIP_TEMP_TABLE flag is on.

Problem was as part of SET PASSWORD FOR ROOT, mysql.user table changed
compared to how it was originally created. (plugin changed value)

The problem occurred because the Spider node was incorrectly handling
timestamp values sent to and received from the data nodes.

The problem has been corrected as follows:
- Added logic to set and maintain the UTC time zone on the data nodes.
  To prevent timestamp ambiguity, it is necessary for the data nodes to use
  a time zone such as UTC which does not have daylight savings time.
- Removed the spider_sync_time_zone configuration variable, which did not
  solve the problem and which interfered with the solution.
- Added logic to convert to the UTC time zone all timestamp values sent to
  and received from the data nodes.  This is done for both unique and
  non-unique timestamp columns.  It is done for WHERE clauses, applying to
  SELECT, UPDATE and DELETE statements, and for UPDATE columns.
- Disabled Spider's use of direct update when any of the columns to update is
  a timestamp column.  This is necessary to prevent false duplicate key value
  errors.
- Added a new test spider.timestamp to thoroughly test Spider's handling of
  timestamp values.

Author:
  Jacob Mathew.

Reviewer:
  Kentoku Shiba.

Cherry-Picked:
  Commit 97cc9d34 on branch bb-10.3-MDEV-16246

The parameter innodb_lock_schedule_algorithm was introduced in
MariaDB Server 10.1.19, 10.2.13, 10.3.4 as part of MDEV-11039.
In MariaDB 10.1, the default value of the parameter is 'fcfs',
that is, the existing algorithm is used by default. But in
later versions of MariaDB Server, the parameter was 'vats',
enabling the new algorithm.

Because the new algorithm is triggering a debug assertion failure
that suggests corruption of the transactional lock data structures,
we will revert to the old algorithm by default until we have
resolved the problem.

Problem:
========
Truncate operation holds MDL on the table (t1) and tries to
acquire InnoDB dict_operation_lock. Purge holds dict_operation_lock
and tries to acquire MDL on the table (t1) to evaluate virtual
column expressions for indexed virtual columns.
It leads to deadlock of purge and truncate table (DDL).

Solution:
=========
If purge tries to acquire MDL on the table then it should do the following:

i) Purge should release all innodb latches (including dict_operation_lock)
before acquiring metadata lock on the table.

ii) After acquiring metadata lock on the table, it should check whether the
table was dropped or renamed. If the table is dropped then purge should
ignore the undo log record. If the table is renamed then it should
release the old MDL and acquire MDL on the new name.

iii) Once purge acquires MDL, it should use the SQL table handle for all
the remaining virtual index for the purge record.

purge_node_t: Introduce new virtual column information to know whether
the MDL was acquired successfully.

This is joint work with Marko Mäkelä.

Add an explicit redo log flush. In this test
innodb_flush_log_at_trx_commit was 2 by default.
It is also possible that this failure occurs because of MDEV-15740.

1. The changed variant did not fail without the patch for MDEV-16629
while the original test case did fail.
2. In any case the test case should go to cte_recursive_not_embedded.test
that was not created yet.

At the end of a test, 'connection default' should be in a usable state.
This was not the case, because there was a preceding 'send' without a
'reap'. If 'reap' was added, an error would be reported because the
server was restarted after the 'send'. It is easiest to 'send' from a
separate connection and do the restart from 'connection default'.

Make dict_table_t::n_ref_count private, and protect it with
a combination of dict_sys->mutex and atomics. We want to be
able to invoke dict_table_t::release() without holding
dict_sys->mutex.