This is a follow-up task to MDEV-12026, which introduced
innodb_checksum_algorithm=full_crc32 and a simpler page format.
MDEV-12026 did not enable full_crc32 for page_compressed tables,
which we will be doing now.

This is joint work with Thirunarayanan Balathandayuthapani.

For innodb_checksum_algorithm=full_crc32 we change the
page_compressed format as follows:

FIL_PAGE_TYPE: The most significant bit will be set to indicate
page_compressed format. The least significant bits will contain
the compressed page size, rounded up to a multiple of 256 bytes.

The checksum will be stored in the last 4 bytes of the page
(whether it is the full page or a page_compressed page whose
size is determined by FIL_PAGE_TYPE), covering all preceding
bytes of the page. If encryption is used, then the page will
be encrypted between compression and computing the checksum.
For page_compressed, FIL_PAGE_LSN will not be repeated at
the end of the page.

FSP_SPACE_FLAGS (already implemented as part of MDEV-12026):
We will store the innodb_compression_algorithm that may be used
to compress pages. Previously, the choice of algorithm was written
to each compressed data page separately, and one would be unable
to know in advance which compression algorithm(s) are used.

fil_space_t::full_crc32_page_compressed_len(): Determine if the
page_compressed algorithm of the tablespace needs to know the
exact length of the compressed data. If yes, we will reserve and
write an extra byte for this right before the checksum.

buf_page_is_compressed(): Determine if a page uses page_compressed
(in any innodb_checksum_algorithm).

fil_page_decompress(): Pass also fil_space_t::flags so that the
format can be determined.

buf_page_is_zeroes(): Check if a page is full of zero bytes.

buf_page_full_crc32_is_corrupted(): Renamed from
buf_encrypted_full_crc32_page_is_corrupted(). For full_crc32,
we always simply validate the checksum to the page contents,
while the physical page size is explicitly specified by an
unencrypted part of the page header.

buf_page_full_crc32_size(): Determine the size of a full_crc32 page.

buf_dblwr_check_page_lsn(): Make this a debug-only function, because
it involves potentially costly lookups of fil_space_t.

create_table_info_t::check_table_options(),
ha_innobase::check_if_supported_inplace_alter(): Do allow the creation
of SPATIAL INDEX with full_crc32 also when page_compressed is used.

commit_cache_norebuild(): Preserve the compression algorithm when
updating the page_compression_level.

dict_tf_to_fsp_flags(): Set the flags for page compression algorithm.
FIXME: Maybe there should be a table option page_compression_algorithm
and a session variable to back it?

Whenever we are reading the first page of a data file, we may have to
adjust the provisionally created fil_space_t::flags to match what is
actually inside the data files. In this way, we will never accidentally
change the format of a data file.

fil_node_t::read_page0(): After validating the FIL_SPACE_FLAGS,
always assign them to space->flags.

btr_root_adjust_on_import(), Datafile::validate_to_dd(),
fil_space_for_table_exists_in_mem(): Adapt to the fix
in fil_node_t::read_page0().

fsp_flags_try_adjust(): Skip the adjustment if full_crc32 is being
used. This adjustment was introduced in MDEV-11623 for upgrading
from MariaDB 10.1.0 to 10.1.20, which used an accidentally changed
format of FIL_SPACE_FLAGS. MariaDB before 10.4.3 never set the
flag that now indicates the full_crc32 format.

MODIFY COLUMN

Do Not create prefix field for long unique key

Use table->record[0] for ha_index_read_map so that vfield gets automatically
be updated.

Follow-up fix for commit b234f810

* Disable `wsrep_sync_wait` before killing galera node which may be
  non-primary (kill_galera.inc causes lock wait timeouts  due to
  wsrep_sync_wait)

* Remove unnecessary `--sleep 1`

* Replace ```SELECT COUNT(*) = 0 ...``` with
  ```SELECT COUNT(*) `expect 0` ...```

MDEV-18809 Server crash in fields_in_hash_keyinfo or Assertion `key_info->key_part->field->flags & (1<< 30)' failed in setup_keyinfo_hash

Move calling setup_keyinfo_hash until all continue is exhausted.
And also call re_setup_keyinfo_hash for goto err.

Don't add long key into share->keys_for_keyread

* MDEV-16509 Improve wsrep commit performance with binlog disabled

Release commit order critical section early after trx_commit_low() if
binlog is not transaction coordinator. In order to avoid two phase commit,
binlog_hton is not registered for THD during IO_CACHE population.

Implemented a test which verifies that the transactions release
commit order early.

This optimization will change behavior during recovery as the commit
is not two phase when binlog is off. Fixed and recorded wsrep-recover-v25
and wsrep-recover to match the behavior.

* MDEV-18730 Ordering for wsrep binlog group commit

Previously out of order execution was allowed for wsrep commits.
Established proper ordering by populating wait_for_commit
for every wsrep THD and making group commit leader to wait for
prior commits before proceeding to trx_group_commit_leader().

* MDEV-18730 Added a test case to verify correct commit ordering

* MDEV-16509, MDEV-18730 Review fixes

Use WSREP_EMULATE_BINLOG() macro to decide if the binlog_hton
should be registered. Whitespace/syntax fixes and cleanups.

* MDEV-16509 Require binlog for galera_var_innodb_disallow_writes test

If the commit to InnoDB is done in one phase, the native InnoDB behavior
is that the transaction is committed in memory before it is persisted to
disk. This means that the innodb_disallow_writes=ON may not prevent
transaction to become visible to other readers before commit is completely
over. On the other hand, if the commit is two phase (as it is with binlog),
the transaction will be blocked in prepare phase.

Fixed the test to use binlog, which enforces two phase commit, which
in turn makes commit to block before the changes become visible to
other connections. This guarantees that the test produces expected
result.

           depends on uninitialized value

This problem most probably was resolved by the patch for MDEV-18816.
This commit adds only the test case from the bug entry.

appeared to leave warning due to expectable full exit of IO threads.
Fixed to ignore warnings at the final START SLAVEs.

Note: any error from row_undo() will lead to the server being
killed by InnoDB.

The warning was removed as this is a common case that happens if the table
was dropped and later created during the same checkpoint or if there was
a bulk insert done on an empty table.

 `!pk->has_virtual()' failed in instant_alter_column_possible upon adding key

Hash key can't be primary key.

As MDEV-18799 fixes these mdevs so adding test cases for these mdevs.

Restore table->key_info after calling setup_keyinfo_hash in
mysql_prepare_alter_table.

Fixed the typo and updated the test results

Fixed, now server start with optimizer_trace enabled would not show the warning

attempt to drop BLOB with long index

Restore long table->key_info, So that in the case of unsuccessful table
alter table->key_info should have a correct value. In the case of successful
table alter old table is flushed so that is why we don't see this error in
the case of successful alter.

The patches features an optional shutdown behavior to hold on until
after all connected slaves have been sent the last binlogged event.
The connected slave is one whose START SLAVE has been acknowledged and
that was not stopped since that though it could be technically
reconnecting in background.

The solution therefore disallows killing the dump thread until is has
found EOF of the latest binlog file.  It is up to the shutdown
requester (DBA) to set up a sufficiently large shutdown timeout value
for shudown to wait patiently until lagging behind slaves have been
synchronized. On the other hand if a specific slave needs exclusion
from synchronization the DBA would have to stop it manually which
would terminate its dump thread.

`mysqladmin shutdown' is extended with a `--wait_for_all_slaves' option
which translates to `SHUTDOW WAIT FOR ALL SLAVES' sql query
to enable the feature on the client side.

The patch also performs a small refactoring of the server shutdown
around close_connections() to introduce kill thread phases which
are two as of current.

In 10.3, all records will be processed by purge due to MDEV-12288.
But, the insert undo records do not contain a transaction identifier.

row_purge_parse_undo_rec(): Use node->trx_id=TRX_ID_MAX for the
insert undo records. We cannot skip table lookups for these records
after DISCARD TABLESPACE other than by 'detaching' the table from
the undo logs by updating SYS_TABLES.ID on both DISCARD TABLESPACE
and IMPORT TABLESPACE.

Also, remove a redundant condition that was introduced
in the merge commit 814205f3.

recv_parse_log_recs(): Do not compare type if ptr==end_ptr
(we have reached the end of the redo log parsing buffer),
because it will not have been correctly initialized in that case.

GCC 6 and later can optimize away the memset() that is part of
mem_heap_zalloc() in a placement new call. So, instead of relying
on that kind of initialization, explicitly initialize the necessary
fields in the constructors.

que_common_t::que_common_t(): Initialize more fields in the
default constructor.

purge_vcol_info_t::purge_vcol_info_t(): Initialize all fields in
the default constructor.

purge_node_t::purge_node_t(): Initialize all necessary fields.

Reference:

    https://gcc.gnu.org/bugzilla/show_bug.cgi?id=71388

    https://gcc.gnu.org/ml/gcc/2016-02/msg00207.html

row_merge_create_fts_sort_index(): Initialize dict_col_t in
an unambiguous way. GCC 6 and later appear to be able to optimize
away the memset() that is part of mem_heap_zalloc() in the
placement new call. Let us avoid using placement new in order
to ensure that the objects will actually be initialized.

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=71388

https://gcc.gnu.org/ml/gcc/2016-02/msg00207.html

While the latter reference hints that the optimization is only
applicable to non-POD types (and dict_col_t does not define
any member functions before 10.2), it is most consistent to
use the same initialization across all versions.

MDEV-17070 Table corruption or Assertion `table->file->stats.records > 0 || error' or Assertion `!is_set() || (m_status == DA_OK_BULK && is_bulk_op())' failed upon actions on temporary table

This was caused by a combination of factors:
* MyISAM/Aria temporary tables historically never saved the state
  to disk (MYI/MAI), because the state never needed to persist
* certain ALTER TABLE operations modify the original TABLE structure
  and if they fail, the original table has to be reopened to
  revert all changes (m_needs_reopen=1)

as a result, when ALTER fails and MyISAM/Aria temp table gets reopened,
it reads the stale state from the disk.

As a fix, MyISAM/Aria tables now *always* write the state to disk
on close, *unless* HA_EXTRA_PREPARE_FOR_DROP was done first. And
the server now always does HA_EXTRA_PREPARE_FOR_DROP before dropping
a temporary table.

JSON_ARRAY and JSON_OBJECT functions with no arguments now get the
connection charset. Item_func_convert_charset returns the correct
is_json() flag.

ALTER TABLE ... ADD FOREIGN KEY may trigger assertion failure when
it has LOCK=EXCLUSIVE clause or concurrent FLUSH TABLES is being
executed.

In both cases being altered table is marked as flushed, which forces
subsequent attempt to open parent table to re-open. Which in turn is
not allowed while transaction is running.

Rather than opening parent table, just take appropriate MDL lock.

Also removed table_already_fk_prelocked() check: MDL itself has much
better methods to handle duplicate locks. E.g. the former won't acquire
MDL_SHARED_NO_WRITE if it already has MDL_SHARED_READ.

read_command_buf was not freed at exit, which could cause a warning from
valgrind