Follow-up to MDEV-13570 Assertion failure !srv_read_only_mode in --innodb-read-only shutdown when buf_resize_thread is active

logs_empty_and_mark_files_at_shutdown(): Actually skip the debug assertion
when the buf_resize_thread is active. The previous fix skipped the
debug assertion failure when buf_dump_thread is active. Both these
threads are created also in innodb_read_only mode. Depending on how
fast these threads react to the shutdown signal, the debug assertion
could be triggered.

There is no impact on non-debug servers, and very little impact on
debug servers either, because in innodb_read_only shutdown, no InnoDB
files will need to be written.

char* parameter is expected by the message ER_KEY_COLUMN_DOES_NOT_EXITS,
thus pass char*, rather than LEX_STRING.

ERROR_FILE_SYSTEM_LIMITATION was seen by support when backing up large
file. However mariabackup error message was not very helpful,
since it mapped the error to generic catch-all EINVAL.

With the patch, ERROR_FILE_SYSTEM_LIMITATION will be mapped to more
appropriate EFBIG. Also add mapping from ERROR_NO_SYSTEM_RESOURCES
to ENOMEM.

CURRENT_TEST: galera.galera_kill_applier
mysqltest: At line 14: query 'KILL $applier_thread' failed with wrong
errno 1064: 'You have an error in your SQL syntax; check the manual
that corresponds to your MariaDB server version for the right syntax
to use near '' at line 1', instead of 1095...

These parameters and associated code is to be removed in 10.3.
Users can use innodb-page-cleaners > 1 instead.

MDEV-12741: innodb.ibuf_not_empty failed in buildbot with "InnoDB: Trying to do I/O to a tablespace which does not exist"

Background thread is doing ibuf merge, in buf0rea.cc buf_read_ibuf_merge_pages().
It first tries to get page_size and if space is not found it deletes them, but
as we do not hold any mutexes, space can be marked as stopped between that
and buf_read_page_low() for same space. This naturally leads seen error
message on log.

buf_read_page_low(): Add parameter ignore_missing_space = false that
is passed to fil_io()

buf_read_ibuf_merge_pages(): call buf_read_page_low with
ignore_missing_space = true, this function will handle missing
space error code after buf_read_page_low returns.

fil_io(): if ignore_missing_space = true do not print error
message about trying to do I/0 for missing space, just return
correct error code that is handled later.

Make rocksdb.bloomfilter* tests 1.8x faster by doing loading in bulk

There is a race condition in InnoDB startup. A number of
fil_crypt_thread are created by fil_crypt_threads_init(). These threads
may call btr_scrub_complete_space() before btr_scrub_init() was called.
Those too early calls would be accessing an uninitialized scrub_stat_mutex.

innobase_start_or_create_for_mysql(): Invoke btr_scrub_init() before
fil_crypt_threads_init().

fil_crypt_complete_rotate_space(): Only invoke btr_scrub_complete_space()
if scrubbing is enabled. There is no need to update the statistics if
it is not enabled.

ATTRIBUTE_NORETURN is supported on all platforms (MSVS and GCC-like).
It declares that a function will not return; instead, the thread or
the whole process will terminate.

ATTRIBUTE_COLD is supported starting with GCC 4.3. It declares that
a function is supposed to be executed rarely. Rarely used error-handling
functions and functions that emit messages to the error log should be
tagged such.

galera.galera_kill_applier: Make the test less likely to fail
by adding sleep time.

galera.query_cache: Remove data truncation.

Part of the test file looks like it has been misinterpreted as latin1
and wrongly converted to UTF-8 encoding. In MariaDB 10.1, the server would
only warn about data truncation and not issue an error. 10.2 is stricter.
(The test should be carefully reviewed if it really makes sense.)

For running the Galera tests, the variable my_disable_leak_check
was set to true in order to avoid assertions due to memory leaks
at shutdown.

Some adjustments due to MDEV-13625 (merge InnoDB tests from MySQL 5.6)
were performed. The most notable behaviour changes from 10.0 and 10.1
are the following:

* innodb.innodb-table-online: adjustments for the DROP COLUMN
behaviour change (MDEV-11114, MDEV-13613)

* innodb.innodb-index-online-fk: the removal of a (1,NULL) record
from the result; originally removed in MySQL 5.7 in the
Oracle Bug #16244691 fix
https://github.com/mysql/mysql-server/commit/377774689bf6a16af74182753fe950d514c2c6dd

* innodb.create-index-debug: disabled due to MDEV-13680
(the MySQL Bug #77497 fix was not merged from 5.6 to 5.7.10)

* innodb.innodb-alter-autoinc: MariaDB 10.2 behaves like MySQL 5.6/5.7,
while MariaDB 10.0 and 10.1 assign different values when
auto_increment_increment or auto_increment_offset are used.
Also MySQL 5.6/5.7 exhibit different behaviour between
LGORITHM=INPLACE and ALGORITHM=COPY, so something needs to be tested
and fixed in both MariaDB 10.0 and 10.2.

* innodb.innodb-wl5980-alter: disabled because it would trigger an
InnoDB assertion failure (MDEV-13668 may need additional effort in 10.2)

After MDEV-13583: Improvements for MTR rebootstrap introduced in
MDEV-12042 bootsrap correctly creates mysql/innodb_table_stats
and mysql/innodb_index_stats InnoDB tables before innodb_encryption
test starts. These tables are also encrypted or decrypted, thus
we need to wait also these tables (if not we could randomly
get different results as system tablespace and these tables
are encrypted or decrypted in parallel).

Problem was incorrect definition of wsrep_recovery,
trx_sys_update_wsrep_checkpoint and
trx_sys_read_wsrep_checkpoint functions causing
innodb_plugin not to load as there was undefined symbols.

Fixes also MDEV-13488: InnoDB writes CRYPT_INFO even though
encryption is not enabled.

Fixes also MDEV-13093: Leak of Datafile::m_crypt_info on
shutdown after failed startup.

Problem was that we created encryption metadata (crypt_data) for
system tablespace even when no encryption was enabled and too early.
System tablespace can be encrypted only using key rotation.

Test innodb-key-rotation-disable, innodb_encryption, innodb_lotoftables
require adjustment because INFORMATION_SCHEMA INNODB_TABLESPACES_ENCRYPTION
contain row only if tablespace really has encryption metadata.

xb_load_single_table_tablespace(): Do not call
fil_space_destroy_crypt_data() any more, because Datafile::m_crypt_data
has been removed.

fil_crypt_realloc_iops(): Avoid divide by zero.

fil_crypt_set_thread_cnt(): Set fil_crypt_threads_event if
encryption threads exist. This is required to find tablespaces
requiring key rotation if no other changes happen.

fil_crypt_find_space_to_rotate(): Decrease the amount of time waiting
when nothing happens to better enable key rotation on startup.

fil_ibd_open(), fil_ibd_load(): Load possible crypt_data from first
page.

class Datafile, class SysTablespace : remove m_crypt_info field.

Datafile::get_first_page(): Return a pointer to first page buffer.

fsp_header_init(): Write encryption metadata to page 0 only if
tablespace is encrypted or encryption is disabled by table option.

i_s_dict_fill_tablespaces_encryption(): Skip tablespaces that do not
contain encryption metadata. This is required to avoid too early
wait condition trigger in encrypted -> unencrypted state transfer.

with Visual Studio

simplify logic, VS generator seems to have problems if generated file
(with ADD_CUSTOM_COMMAND) depends on another generated file.

So, the fix is just to have mysqld_lib.{def,lib,exp} to be generated in a
single ADD_CUSTOM_COMMAND rather than two steps.

allow to build with -DPLUGIN_INNOBASE=NO

Update wait_timeout.test to add test case for this.

wsrep_drop_table_query(): Remove the definition of this ununsed function.

row_upd_sec_index_entry(), row_upd_clust_rec_by_insert():
Evaluate the simplest conditions first. The merge could have slightly
hurt performance by causing extra calls to wsrep_on().

Make rocksdb.cardinality test faster (77 -> 42 sec with --mem) by
loading records in batches.
(loading everything as one bulk load batch will remove the purpose
 of the test)

This is not affecting correctness; delete NULL is a valid operation.

recv_find_max_checkpoint(): Refer to MariaDB 10.2.2 instead of
MySQL 5.7.9. Do not hint that a binary downgrade might be possible,
because there are many changes in InnoDB 5.7 that could make
downgrade impossible: a column appended to SYS_INDEXES, added
SYS_* tables, undo log format changes, and so on.

It is possible that a .git directory isn't definitive on the
existance of a working git tree. A git worktree over sshfs
for instance will fail to build unless the main repository
also exists in the same directory. Adding this extra test
will make the detection that little more ruggardised for these
odd build environments.
Signed-off-by: Daniel Black <daniel.black@au.ibm.com>

Import the MySQL 5.6 addition from innodb.create-index to a new debug-only
test, innodb.create-index-debug. The existing test innodb.create-index
also runs on a debug server.

FIXME: MDEV-13668 InnoDB unnecessarily rebuilds table

FIXME: MDEV-13671 InnoDB should use case-insensitive column name comparisons
like the rest of the server

FIXME: MDEV-13640 / Properly fix MDEV-9469 'Incorrect key file' on ALTER TABLE

FIXME: investigate result difference in innodb.innodb-alter-autoinc
and ensure that MariaDB does the right thing with auto_increment_increment
and auto_increment_offset, for both ALGORITHM=INPLACE and ALGORITHM=COPY
(Oracle MySQL behaviour differs between those two).

Import some ALTER TABLE test cases from MySQL 5.6 without modification.
The adjustments will be in a separate commit.

Fixes also MDEV-13488: InnoDB writes CRYPT_INFO even though
encryption is not enabled.

Problem was that we created encryption metadata (crypt_data) for
system tablespace even when no encryption was enabled and too early.
System tablespace can be encrypted only using key rotation.

Test innodb-key-rotation-disable, innodb_encryption, innodb_lotoftables
require adjustment because INFORMATION_SCHEMA INNODB_TABLESPACES_ENCRYPTION
contain row only if tablespace really has encryption metadata.

fil_crypt_set_thread_cnt: Send message to background encryption threads
if they exits when they are ready. This is required to find tablespaces
requiring key rotation if no other changes happen.

fil_crypt_find_space_to_rotate: Decrease the amount of time waiting
when nothing happens to better enable key rotation on startup.

fsp_header_init: Write encryption metadata to page 0 only if tablespace is
encrypted or encryption is disabled by table option.

i_s_dict_fill_tablespaces_encryption : Skip tablespaces that do not
contain encryption metadata. This is required to avoid too early
wait condition trigger in encrypted -> unencrypted state transfer.

open_or_create_data_files: Do not create encryption metadata
by default to system tablespace.

Assertions failed due to incorrect handling of the --tc-heuristic-recover
option when InnoDB is in read-only mode either due to innodb_read_only=1
or innodb_force_recovery>3. InnoDB failed to refuse a XA COMMIT or
XA ROLLBACK operation, and there were errors in the error handling in
the upper layer.

This was fixed by making InnoDB XA operations respect the
high_level_read_only flag. The InnoDB part of the fix and
parts of the test main.tc_heuristic_recover were provided
by Marko Mäkelä.

LOCK_log mutex lock/unlock had to be added to fix MDEV-13438.
The measure is confirmed by mysql sources as well.

For testing of the conflicting option combination, mysql-test-run is
made to export a new $MYSQLD_LAST_CMD. It holds the very last value
generated by mtr.mysqld_start().  Even though the options have been
also always stored in $mysqld->{'started_opts'} there were no access
to them beyond the automatic server restart by mtr through the expect
file interface.

Effectively therefore $MYSQLD_LAST_CMD represents a more general
interface to $mysqld->{'started_opts'} which can be used in wider
scopes including server launch with incompatible options.

Notice another existing method to restart the server with incompatible
options relying on $MYSQLD_CMD is is aware of $mysqld->{'started_opts'}
(the actual options that the server is launched by mtr). In order to use
this method they would have to be provided manually.

NOTE: When merging to 10.2, the file search_pattern_in_file++.inc
should be replaced with the pre-existing search_pattern_in_file.inc.

If this variable is set, skip actual AWS calls, and fake/mock
both generation and encryption of the keys.

The advantage of having a mock mode is that more aws_key_management tests
can be enabled on buildbot.

- make re-bootstrap run with all extra options, not only InnoDB ones
- re-use previously created bootstrap.sql
- add --console
- fix debian patch to keep it applicable