The InnoDB change buffer (ibuf.index, stored in the system tablespace)
and the change buffer bitmaps in persistent tablespaces could get out
of sync with each other: According to the bitmap, no changes exist for
a page, while there actually exist buffered entries in ibuf.index.

InnoDB performs lazy deletion of buffered changes. When a secondary
index leaf page is freed (possibly as part of DROP INDEX), any
buffered changes will not be deleted. Instead, they would be deleted
on a subsequent buf_page_create_low().

One scenario where InnoDB failed to delete buffered changes is
as follows:
1. Some changes were buffered for a secondary index leaf page.
2. The index page had been freed.
3. ibuf_read_merge_pages() invoked ibuf_merge_or_delete_for_page(),
which noticed that the page had been freed, and reset the change buffer
bits, but did not delete the records from ibuf.index.
4. The index page was reallocated for something else.
5. The index page was removed from the buffer pool.
6. Some changes were buffered for the newly created page.
7. Finally, the buffered changes from both 1. and 6. were merged.
8. The index is corrupted.

An alternative outcome is:
4. Shutdown with innodb_fast_shutdown=0 gets into an infinite loop.

An alternative scenario is:
3. ibuf_set_bitmap_for_bulk_load() reset the IBUF_BITMAP_BUFFERED bit
but did not delete the ibuf.index records for that page number.

The shutdown hang was already once fixed in
commit d7a24017, refactored for
10.5 in commit 77e8a311 and
disabled in commit 310dff5d
due to corruption.

We will fix this as follows:

ibuf_delete_recs(): Delete all ibuf.index entries for the specified page.

ibuf_merge_or_delete_for_page(): When the change buffer bitmap bits
were set and the page had been freed, and the page does not belong
to ibuf.index itself, invoke ibuf_delete_recs(). This prevents the
corruption from occurring when a DML operation is allocating a
previously freed page for which changes had been buffered.

ibuf_set_bitmap_for_bulk_load(): When the change buffer bitmap bits
were set, invoke ibuf_delete_recs(). This prevents the corruption
from occurring when CREATE INDEX is reusing a previously freed page.

ibuf_read_merge_pages(): On slow shutdown, remove the orphan records
by invoking ibuf_delete_recs(). This fixes the hang when the change
buffer had become corrupted. We also remove the dops[] accounting,
because nothing can monitor it during shutdown. We invoke
ibuf_delete_recs() if:
(a) buf_page_get_gen() failed to load the page or merge changes
(b) the page is not a valid index leaf page
(c) the page number is out of tablespace bounds

srv_shutdown(): Invoke ibuf_max_size_update(0) to ensure that
the race condition that motivated us to disable the code in
ibuf_read_merge_pages() in commit 310dff5d
is no longer possible. That is, during slow shutdown, both the
rollback of transactions and the purge of history will return
early from ibuf_insert_low().

ibuf_merge_space(), ibuf_delete_for_discarded_space(): Cleanup:
Do not allocate a memory heap.

This was implemented by Thirunarayanan Balathandayuthapani
and tested with innodb_change_buffering_debug=1 by Matthias Leich.

fil_name_process(): If fil_ibd_load() returns FIL_LOAD_INVALID,
display the file name and the tablespace identifier.

recv_sys_t::recover_deferred(): If the *.ibd file already exists,
adjust the size to the tablespace metadata. It could be that
in a multi-batch recovery, we will initially recover an all-zero
*.ibd file to a smaller size, and then a fatal error would be
reported during the last recovery batch.

This bug could be worked around by executing the recovery again.
During the initial (failed) recovery attempt, something should have
been written to the first page of the file and the file size should
be recovered by fil_node_t::read_page0().

During crash recovery, recv_sys.apply(true) invokes
mlog_init.mark_ibuf_exist(), which in turn may invoke
recv_sys.apply(true) via the buf_flush_sync() call in
buf_page_get_low(). The simplest fix is to disable the
innodb_change_buffering_debug=1 instrumentation
during crash recovery.

# Conflicts:
#	sql/sql_connect.cc

Remove DBUG_ASSERT that depends on timing.

Remove DBUG_ASSERT, that depends on timing/scheduling to succeed.

The log overwrite warnings are not being reliably emitted in all
debug-instrumented environments. It may be related to the
scheduling of some InnoDB internal activity, such as the purging
of committed transaction history.

btr_cur_t: Zero-initialize all fields in the default constructor.

btr_cur_t::index: Remove; it duplicated page_cur.index.

Many functions: Remove arguments that were duplicating
page_cur_t::index and page_cur_t::block.

page_cur_open_level(), btr_pcur_open_level(): Replaces
btr_cur_open_at_index_side() for dict_stats_analyze_index().
At the end, release all latches except the dict_index_t::lock
and the buf_page_t::lock on the requested page.

dict_stats_analyze_index(): Rely on mtr_t::rollback_to_savepoint()
to release all uninteresting page latches.

btr_search_guess_on_hash(): Simplify the logic, and invoke
mtr_t::rollback_to_savepoint().

We will use plain C++ std::vector<mtr_memo_slot_t> for mtr_t::m_memo.
In this way, we can avoid setting mtr_memo_slot_t::object to nullptr
and instead just remove garbage from m_memo.

mtr_t::rollback_to_savepoint(): Shrink the vector. We will be needing this
in dict_stats_analyze_index(), where we will release page latches and
only retain the index->lock in mtr_t::m_memo.

mtr_t::release_last_page(): Release the last acquired page latch.
Replaces btr_leaf_page_release().

mtr_t::release(const buf_block_t&): Release a single page latch.
Used in btr_pcur_move_backward_from_page().

mtr_t::memo_release(): Replaced with mtr_t::release().

mtr_t::upgrade_buffer_fix(): Acquire a latch for a buffer-fixed page.
This replaces the double bookkeeping in btr_cur_t::open_leaf().

Reviewed by: Vladislav Lesin

MDEV-25625 Test sys_vars.wsrep_on_without_provider fails: mysqltest: At line 8: query 'SET GLOBAL wsrep_on=ON' failed with wrong errno 1193: 'Unknown system variable 'wsrep_on'', instead of 1210...

btr_cur_t::open_leaf(): Replaces btr_cur_open_at_index_side() for
most calls, except dict_stats_analyze_index(), which is the only
place where we need to open a page at the non-leaf level.
Use btr_block_get() for better error handling.

Also, use the enumeration type btr_latch_mode wherever possible.

Reviewed by: Vladislav Lesin

The Boolean field btr_pcur_t::old_stored mostly duplicates old_rec.
Let us remove it.

btr_cur_search_to_nth_level(): Simply acquire a latch on the already
buffer-fixed page. There is no need to release the buffer-fix and
re-lookup the page.

btr_blob_log_check_t::check(): Acquire an index U-latch upfront and
pass the flag BTR_MODIFY_LEAF_ALREADY_LATCHED to the search.

If a log checkpoint occurs at the end LSN of mtr.commit_shrink(space)
in trx_purge_truncate_history(), then recovery may fail because
it could try to apply too old log records to too old copies of
undo log pages. This was repeated with the following test:

./mtr innodb.undo_log_truncate,4k,strict_full_crc32

recv_sys_t::trim(): Move some code to the caller.

recv_sys_t::apply(): For undo tablespace truncation, discard
all old redo log for the undo tablespace, and then truncate
the file to the desired size.

Tested by: Matthias Leich

ibuf.size, ibuf.max_size: Changed the type to Atomic_relaxed<ulint>
in order to fix some (not all) race conditions.

ibuf_contract(): Renamed from ibuf_merge_pages(ulint*).

ibuf_merge(), ibuf_merge_all(): Removed.

srv_shutdown(): Invoke log_free_check() and ibuf_contract(). Even though
ibuf_contract() is not writing anything, it will trigger calls of
ibuf_merge_or_delete_for_page(), which will write something. Because
we cannot invoke log_free_check() at that low level, we must invoke
it at the high level.

srv_shutdown_print(): Replaces srv_shutdown_print_master_pending().
Report progress and remaining work every 15 seconds. For the
change buffer merge, the remaining work is indicated by ibuf.size.

row_check_index(): Treat secondary indexes of temporary tables as if
SET TRANSACTION ISOLATION LEVEL READ UNCOMMITTED
is in effect. That is, only consider the delete-mark and nothing else.

- Changed the page0 perl corruption code in test case to avoid
the bogus error in windows

The InnoDB write-ahead log ib_logfile0 is of fixed size,
specified by innodb_log_file_size. If the tail of the log
manages to overwrite the head (latest checkpoint) of the log,
crash recovery will be broken.

Let us clarify the messages about this, including adding
a message on the completion of a log checkpoint that notes
that the dangerous situation is over.

To reproduce the dangerous scenario, we will introduce the
debug injection label ib_log_checkpoint_avoid_hard, which will
avoid log checkpoints even harder than the previous
ib_log_checkpoint_avoid.

log_t::overwrite_warned: The first known dangerous log sequence number.
Set in log_close() and cleared in log_write_checkpoint_info(),
which will output a "Crash recovery was broken" message.

srv_shutdown(): Do not call log_free_check(), because it will now
be repeatedly called by ibuf_merge_all(). Do not call
srv_sync_log_buffer_in_background(), because we do not actually care
about durability during shutdown. Log writes will already be triggered
by buf_flush_page_cleaner() for writing back modified pages, possibly by
log_free_check().

logs_empty_and_mark_files_at_shutdown(): Clean up a condition.
This function is the caller of srv_shutdown(), and it will ensure that
the log and the buffer pool will be in clean state before shutdown.

This bug was previously fixed in 10.6.11 by:
   MDEV-28327 InnoDB persistent statistics fail to update after bulk insert

Adding MTR tests only.

Also, fixing the old test for MDEV-28327 to make "mtr" reliably pass
with/without --mysqld=--innodb-stats-persistent=0, and with different page sizes,
as suggested by Marko.

* Deb: Handle codename 'n/a' from Debian Sid properly and autobake-deb cleanup

This fixes autobake-deb.sh builds on Sid which was visible as 4 failing
build steps on Salsa-CI.

- In Sid the LSBNAME might evaluate to 'n/a', so accept it as 'Sid' to
  fix builds that failed with error:

    Error - unknown release codename n/a

- Refactor list to have Ubuntu versions first, then Debian, and as last
  the special case of Debian Sid

- Fix minor syntax issues detected by Shellcheck

Also remove useless DEB_HOST_ARCH_CPU check from debian/rules:
* It was never in effect as the 'sed' in autobake-deb.sh cleared it anyway
* The variable name was wrong and always empty
* If variable would have been correct, logic was still reversed

- Define 3h timeout as the default 1h timeout on Gitlab.com (and others)
  is usually not enough for initial (uncached) MariaDB builds.

- Replace Buster to Bookworm/Sid upgrade testing with upgrade inside Buster
  testing as direct upgrades from Stretch to Bullseye and Buster to Bookworm
  are no longer possible due to:

    Bug#993755: libcrypt.so.1: cannot open shared object file when
    upgrading from Stretch to Sid (https://bugs.debian.org/993755)

- Stop ignoring MariaDB.org 10.6 to this version upgrade testing failures
  to reveal bug MDEV-28640. Originally this step was failing as the uring
  dependencies in upstream builders lagged behind and there was nothing
  that needed work, only time time to resolve. Now there is an actual bug
  in packaging that should be visible as a CI failure.

- Stop testing for 'service mysql status' on systems that upgraded from
  MySQL 8.0 to MariaDB.org vended 10.6. Due to some unidentified debian/control
  changes in 10.6 on upstream the upgrade is no longer compatible in
  a way that would maintain the init.d script with name 'mysql'.

- Fix typos where mergers had changed occurrences of 10.5 to 10.6 while
  they intentionally need to be exactly 10.5, otherwise the meaning
  changes.

- Align autopkgtest code with downstream official Debian packaging one.
  This is change is safe on a stable branch because is only affects builds
  and testing, not any actual usage of MariaDB 10.6.

- Standardize on using capitalized 'YES' in CMake build options
  (instead of 'yes' or mixed case)

- Add some comments to better document debian/rules

- Fix typo in Lintian overrides

Ubuntu bug: https://bugs.launchpad.net/ubuntu/+source/mariadb-10.6/+bug/1970634
MariaDB ticket: https://jira.mariadb.org/browse/MDEV-25633

When built with LTO on Ubuntu, MariaDB does not catch an exception when
the uring initialization fails due to a low RLIMIT_MEMLOCK value.

This commit amends the commit 0609b345
to be identical to the one done downstream in Debian:
https://salsa.debian.org/mariadb-team/mariadb-server/-/commit/8d20ca979cf422d3a507283b86c2547d78559179

This way both the inline comments and 'git blame' for this section will
show properly why this is needed, and the fix is one that is fully tested
on Debian and Ubuntu.

Also having this section fully identical in upstream MariaDB and downstream
Debian will make the packaging maintenance easier as 'diff` runs on this
file will not flag this as a difference anymore.

In MDEV-28640 the init script failed to stop/start the MariaDB server
due to missing mysqladmin on the system. This was however very hard to
spot from the console output.

Add an explicit check for the binary the script depends on, and fail
verbosely if the dependency is missing.