This fixes up 4eb8e51c
which missed a second parent.

Some builders in our CI, most notably FreeBSD and IBM AIX, do not support
sparse files. Also, Microsoft Windows requires special means for creating
sparse files. Since these platforms do not run ./mtr --big-test, we will
for now simply move the test to a separate file that requires that option.

This commit contains only a mtr test for reproducing the issue in MDEV-29512
The actual fix will be pushed in wsrep-lib repository

The hanging in MDEV-29512 happens when binlog purging is attempted, and there is
one local BF aborted transaction waiting for commit monitor.

The test will launch two node cluster and enable binlogging with expire log days,
to force binlog purging to happen.
A local transaction is executed so that will become BF abort victim, and has advanced
to replication stage waiting for commit monitor for final cleanup (to mark position in innodb)
after that, applier is released to complete the BF abort and due to binlog configuration,
starting the binlog purging. This is where the hanging would occur, if code is buggy
Reviewed-by: Jan Lindström <jan.lindstrom@mariadb.com>

Test MDEV-26575 fails when it runs after MDEV-25389. This is because
the latter simulates a failure while an applier thread is
created in `start_wsrep_THD()`. The failure was not handled correctly
and would not cleanup the created THD from the global
`server_threads`. A subsequent shutdown would hang and eventually fail
trying to close this THD.
Reviewed-by: Jan Lindström <jan.lindstrom@mariadb.com>

Reviewed-by: Jan Lindström <jan.lindstrom@mariadb.com>

Fix `wsrep_table_accessible_when_detached()` so that commands that
access no tables are rejected while a node is disconnected from a
cluster.
Reviewed-by: Jan Lindström <jan.lindstrom@mariadb.com>

Prevent wsrep files from being installed if WITH_WSREP=OFF.

Reviewed by Daniel Black
Additionally excluded #include wsrep files and galera* files
along with galera/wsrep tests.

mysql-test/include/have_wsrep.inc remainds as its used by
a few isolated tests.
Co-authored-by: Chris Ross <cross2@cisco.com>

recv_log_recover_10_4(): Widen the operand of bitwise and to 64 bits,
so that the upgrade check will work when the redo log record is located
more than 4 gigabytes from the start of the first file.

ibuf_init_at_db_start(): Validate the change buffer root page.
A later version may stop creating a change buffer, and this
validation check will prevent a downgrade from such later versions.

ibuf_max_size_update(): If the change buffer was not loaded, do nothing.

dict_boot(): Merge the local variable "error" to "err". Ignore
failures of ibuf_init_at_db_start() if innodb_force_recovery>=4.

In commit a03dd94b as well as
mysql/mysql-server@6ef8c343445a26aaf9ebd76d72cf57db44b481f5
the iterations were changed so that the variable "scanned"
would remain 0 when the first list item qualifies for eviction.

buf_LRU_free_from_unzip_LRU_list(), buf_LRU_free_from_common_LRU_list():
Increment "scanned" when a block can be freed.

buf_LRU_free_from_common_LRU_list(): Remove a redundant condition.
Whenever this function is invoked, buf_pool.LRU should be nonempty,
hence something should always be scanned.

Thanks to Jean-François Gagné for reporting this.

The very lightest of load would decimate any buffer pool loading
to ~1 page per second. As seen in MDEV-29343 this resulting in
a load taking over an hour on a high end system.

Since MDEV-26547 the fetching is asynchronous, however the loading
has equal access to the IO as the SQL queries.

Signed-off-by: Weijun-H <huangweijun1001@gmail.com>

Signed-off-by: Weijun-H <huangweijun1001@gmail.com>

Problem:
========
There is a race condition in binlog.binlog_checkpoint between the
binlog background thread creating a binlog checkpoint event, and the
connection thread binlogging a query event for creating a table.
Because the test outputs the events for validation, the order
between these two events can be different, resulting in a failed
test.

Solution:
========
Instead of outputting the binlog events, use assert_grep to validate
the content of the binlog is correct.

Reviewed By:
============
Andrei Elkin <andrei.elkin@mariadb.com>

A few query cache functions don't use THD pointer
so its removed from their interface.

Fix the regression introduced in
dfb41fdd.

In the restructure of mysql_rm_table_no_locks the early condition
of !frm_error that enabled non_tmp_table_deleted, and hence the
query cache invalidation, was removed.

The query_cache_invalidate1(thd, dbnorm) called after
mysql_rm_table_no_locks depends on the query cache removal
(for unexamined reasons).

Under DROP DATABASE, in mysql_rm_table_no_locks, dont_log_query
is true preventing the late setting of non_tmp_table_deleted
(which retained one of its purposes as a replication deletion
of temporary tables, but not query cache invalidation).

The non_temp_tables_count however can still be used to invalidate
the query cache.

This is a DELETE only case. Normally this statement doesn't make inserts,
but DELETE ... FOR PORTION changes it. UPDATE and INSERT initializes
autoinc by calling handler::info(HA_STATUS_AUTO). Also myisam and innodb
can lazily initialize it in their update_create_info overrides.

The solution is to initialize autoinc during delete preparation,
if period (DELETE FOR PORTION) is specified.

The initial work has been done by Kento Takeuchi by his PR #2048,
however this commit also holds a few technical modifications by
Nikita Malyavin

The InnoDB change buffer (ibuf.index, stored in the system tablespace)
and the change buffer bitmaps in persistent tablespaces could get out
of sync with each other: According to the bitmap, no changes exist for
a page, while there actually exist buffered entries in ibuf.index.

InnoDB performs lazy deletion of buffered changes. When a secondary
index leaf page is freed (possibly as part of DROP INDEX), any
buffered changes will not be deleted. Instead, they would be deleted
on a subsequent buf_page_create_low().

One scenario where InnoDB failed to delete buffered changes is
as follows:
1. Some changes were buffered for a secondary index leaf page.
2. The index page had been freed.
3. ibuf_read_merge_pages() invoked ibuf_merge_or_delete_for_page(),
which noticed that the page had been freed, and reset the change buffer
bits, but did not delete the records from ibuf.index.
4. The index page was reallocated for something else.
5. The index page was removed from the buffer pool.
6. Some changes were buffered for the newly created page.
7. Finally, the buffered changes from both 1. and 6. were merged.
8. The index is corrupted.

An alternative outcome is:
4. Shutdown with innodb_fast_shutdown=0 gets into an infinite loop.

An alternative scenario is:
3. ibuf_set_bitmap_for_bulk_load() reset the IBUF_BITMAP_BUFFERED bit
but did not delete the ibuf.index records for that page number.

The shutdown hang was already once fixed in
commit d7a24017, refactored for
10.5 in commit 77e8a311 and
disabled in commit 310dff5d
due to corruption.

We will fix this as follows:

ibuf_delete_recs(): Delete all ibuf.index entries for the specified page.

ibuf_merge_or_delete_for_page(): When the change buffer bitmap bits
were set and the page had been freed, and the page does not belong
to ibuf.index itself, invoke ibuf_delete_recs(). This prevents the
corruption from occurring when a DML operation is allocating a
previously freed page for which changes had been buffered.

ibuf_set_bitmap_for_bulk_load(): When the change buffer bitmap bits
were set, invoke ibuf_delete_recs(). This prevents the corruption
from occurring when CREATE INDEX is reusing a previously freed page.

ibuf_read_merge_pages(): On slow shutdown, remove the orphan records
by invoking ibuf_delete_recs(). This fixes the hang when the change
buffer had become corrupted. We also remove the dops[] accounting,
because nothing can monitor it during shutdown. We invoke
ibuf_delete_recs() if:
(a) buf_page_get_gen() failed to load the page or merge changes
(b) the page is not a valid index leaf page
(c) the page number is out of tablespace bounds

srv_shutdown(): Invoke ibuf_max_size_update(0) to ensure that
the race condition that motivated us to disable the code in
ibuf_read_merge_pages() in commit 310dff5d
is no longer possible. That is, during slow shutdown, both the
rollback of transactions and the purge of history will return
early from ibuf_insert_low().

ibuf_merge_space(), ibuf_delete_for_discarded_space(): Cleanup:
Do not allocate a memory heap.

This was implemented by Thirunarayanan Balathandayuthapani
and tested with innodb_change_buffering_debug=1 by Matthias Leich.

This patch adds the correct setting of the "--tls-version" and
"--ssl-verify-server-cert" options in the client-side utilities
such as mysqltest, mysqlcheck and mysqlslap, as well as the correct
setting of the "--ssl-crl" option when executing queries on the
slave side, and also the correct option codes in the "sslopts-logopts.h"
file (in the latter case, incorrect values are not a problem right
now, but may cause subtle test failures in the future, if the option
handling code changes).

fil_name_process(): If fil_ibd_load() returns FIL_LOAD_INVALID,
display the file name and the tablespace identifier.

Also fixes:

MDEV-27768 MDEV-25440: Assertion `(cs->state & 0x20000) == 0' failed in my_strnncollsp_nchars_generic_8bit

The "strnncollsp_nchars" virtual function pointer for tis620_thai_nopad_ci
was incorrectly initialized to a generic function
my_strnncollsp_nchars_generic_8bit(), which crashed on assert.

Implementing a tis620 specific function version.

During crash recovery, recv_sys.apply(true) invokes
mlog_init.mark_ibuf_exist(), which in turn may invoke
recv_sys.apply(true) via the buf_flush_sync() call in
buf_page_get_low(). The simplest fix is to disable the
innodb_change_buffering_debug=1 instrumentation
during crash recovery.

Remove DBUG_ASSERT that depends on timing.

The log overwrite warnings are not being reliably emitted in all
debug-instrumented environments. It may be related to the
scheduling of some InnoDB internal activity, such as the purging
of committed transaction history.

MDEV-25625 Test sys_vars.wsrep_on_without_provider fails: mysqltest: At line 8: query 'SET GLOBAL wsrep_on=ON' failed with wrong errno 1193: 'Unknown system variable 'wsrep_on'', instead of 1210...

If a log checkpoint occurs at the end LSN of mtr.commit_shrink(space)
in trx_purge_truncate_history(), then recovery may fail because
it could try to apply too old log records to too old copies of
undo log pages. This was repeated with the following test:

./mtr innodb.undo_log_truncate,4k,strict_full_crc32

recv_sys_t::trim(): Move some code to the caller.

recv_sys_t::apply(): For undo tablespace truncation, discard
all old redo log for the undo tablespace, and then truncate
the file to the desired size.

Tested by: Matthias Leich

ibuf.size, ibuf.max_size: Changed the type to Atomic_relaxed<ulint>
in order to fix some (not all) race conditions.

ibuf_contract(): Renamed from ibuf_merge_pages(ulint*).

ibuf_merge(), ibuf_merge_all(): Removed.

srv_shutdown(): Invoke log_free_check() and ibuf_contract(). Even though
ibuf_contract() is not writing anything, it will trigger calls of
ibuf_merge_or_delete_for_page(), which will write something. Because
we cannot invoke log_free_check() at that low level, we must invoke
it at the high level.

srv_shutdown_print(): Replaces srv_shutdown_print_master_pending().
Report progress and remaining work every 15 seconds. For the
change buffer merge, the remaining work is indicated by ibuf.size.