MDEV-24781 Assertion `mode == 16 || mode == 12 || fix_block->page.status != buf_page_t::FREED' failed in buf_page_get_low

This is caused by commit 3cef4f8f
(MDEV-515). dict_table_t::clear() frees all the blob during
rollback of bulk insert.But online log tries to read the
freed blob while applying the log. It can be fixed if we
truncate the online log during rollback of bulk insert operation.

The DeadlockChecker expects to be able to freeze the waits-for graph.
Hence, it is best executed somewhere where we are not holding any
additional mutexes.

lock_wait(): Defer the deadlock check to this function, instead
of executing it in lock_rec_enqueue_waiting(), lock_table_enqueue_waiting().

DeadlockChecker::trx_rollback(): Merge with the only caller,
check_and_resolve().

LockMutexGuard: RAII accessor for lock_sys.mutex.

lock_sys.deadlocks: Replaces lock_deadlock_found.

trx_t: Clean up some comments.

InnoDB fails to remove the ahi entries during rollback
of bulk insert operation. InnoDB throws the error when
validates the ahi hash tables. InnoDB should remove
the ahi entries while freeing the segment only during
bulk index rollback operation.

Reviewed-by: Marko Mäkelä

This is caused by commit c92f7e28(MDEV-8139).
InnoDB fails to set the page status as FREED in buffer pool while freeing
the extent.

With the fix, new connection without THDs will now show NULL in
thread_pool_queues.CONNECTION_ID.

filter out tpool

ha_innobase::commit_inplace_alter_table(): Fix a regression that was
introduced in 6d1f1b61 (MDEV-24564).

MDEV-24647 Columnstore storage engine cannot be compiled with mariadb server on aarch64 platform (#1742)

Also update the SEL_ARG graph weight in:
- sel_add()
- SEL_ARG::clone()

Make key_{and,or}_with_limit() to also verify weight for the arguments
(There is no single point to verify SEL_ARG graphs constructed from
conditions that are not AND-OR formulas, so we hope that those are
connected with AND/OR and do it here).

The purpose of the test was to ensure that the SX (update) mode of
index tree and buffer page latches are being used.

The test has become unstable, possibly due to changes related to
buf_pool.mutex and buf_pool.page_hash, or to the use of MDL in the
purge of transaction history.

In 10.6, the test depends on instrumentation that was refactored
or removed in MDEV-24142.

The use of different latching modes can better be indirectly observed
through high-concurrency benchmarks. For MDEV-14637, a performance test
was conducted where the finer-grained latching and
BTR_CUR_FINE_HISTORY_LENGTH were removed. It caused a 20% performance
regression for UPDATE and somewhat smaller for INSERT.

Any new problem with latching granularity should be easily caught by
performance testing, or by stress tests with Random Query Generator.

There are no IO threads anymore.

(Variant #5, full patch, for 10.5)

Do not produce SEL_ARG graphs that would yield huge numbers of ranges.
Introduce a concept of SEL_ARG graph's "weight". If we are about to
produce a graph whose "weight" exceeds the limit, remove the parts
of SEL_ARG graph that represent the biggest key parts. Do so until
the graph's is within the limit.

Includes
- debug code to verify SEL_ARG graph weight
- A user-visible @@optimizer_max_sel_arg_weight to control the optimization
- Logging the optimization into the optimizer trace.

The implementation for MDEV-17048 apperas to be direct copy from mysql version.
The group commit works differently in mariadb and the assert in wsrep_unregister_from_group_commit() is too strict.

The reason is that in: Wsrep_high_priority_service::log_dummy_write_set(), the transaction will undergo full rollback:
    {
      cs.before_rollback();
      cs.after_rollback();
    }

After that, the client's transaction state is set to be:  wsrep::transaction::s_aborted.
The execution then continues execution by:

...
 wsrep_register_for_group_commit(m_thd);
...
 wsrep_unregister_from_group_commit(m_thd);

The bogus assert in wsrep_unregister_from_group_commit() allows only transactions states of :s_ordered_commit or s_aborting.

As the fix, I brought back the same assert as is present in MariaDB 10.4 version.

a. The change makes `mariadb-upgrade` detect if `MYSQL_JSON` data type is needed.
b. Install the data type if it's not installed.
c. Uninstalls the data type once finished.
d. Create `.opt` and `.inc` files `have_type_mysql_json` and adapt the
tests

Reviewed by: vicentiu@mariadb.org

In commit 3cef4f8f (MDEV-515)
we inadvertently broke CREATE TABLE...REPLACE SELECT statements
by wrongly disabling row-level undo logging.

select_create::prepare(): Only invoke extra(HA_EXTRA_BEGIN_ALTER_COPY)
if no special treatment of duplicates is needed.

Ever since commit 007f68c3,
ALTER TABLE no longer invokes handler::open() after
handler::commit_inplace_alter_table().

ha_innobase::reload_statistics(): Reload or recompute statistics
after ALTER TABLE.

innodb_notify_tabledef_changed(): A new function to invoke
ha_innobase::reload_statistics().

handlerton::notify_tabledef_changed(): Add the parameter handler*
so that ha_innobase::reload_statistics() can be invoked.

ha_partition::notify_tabledef_changed(),
partition_notify_tabledef_changed(): Pass through the call
to any partitions or subpartitions.

This is based on code that was supplied by Monty.

The check is added.

During recovery, InnoDB fails if it tries to apply a FREE_PAGE
and WRITE record to the page. InnoDB encryption thread accesses
the freed page and writes redo log for it.

This is similar to commit deadec4e (MDEV-24569)
InnoDB is missing buf_page_free() while freeing the segment.
To avoid accessing of freed page in buffer pool, InnoDB should
mark the pages as FREED while freeing the segment. Also to
avoid reading of freed page, InnoDB should check the
allocation bitmap page.

fseg_free_step(): Mark the page in buffer pool as FREED

fseg_free_step_not_header(): Mark the page in buffer pool as FREED

buf_dump(): Ignore the freed pages while dumping the buffer pool content

fil_crypt_get_page_throttle_func(): Skip the rotation for FREED page
to avoid the assert failure during recovery

fil_crypt_rotate_page(): Skip the rotation for the freed page

Reviewed-by: Marko Mäkelä

Let us prefer member functions to the old C-style accessor functions.
Also, prefer bitwise AND operations for checking multiple flags.

MDEV-24693 LeakSanitizer: detected memory leaks in mem_heap_create_block_func / fts_optimize_create_msg

- This issue is caused by the commit bf1f9b59
(MDEV-24638). Delay the creation of SYNC message in
fts_optimize_request_sync_table. So that InnoDB can avoid creating
the message if the table already has SYNC message in fts_optimize_wq queue

After an ignored INSERT IGNORE statement into an empty table, we would
wrongly use the MDEV-515 table-level undo logging for a subsequent
REPLACE statement.

ha_innobase::reset_template(): Clear m_prebuilt->ins_node->bulk_insert
on every statement boundary.

ha_innobase::start_stmt(): Invoke end_bulk_insert().

ha_innobase::extra(): Avoid accessing m_prebuilt->trx. Do not call
thd_to_trx(). Invoke end_bulk_insert() and try to reset bulk_insert
when changing the REPLACE or IGNORE settings.

trx_mod_table_time_t::WAS_BULK: Use a distinct value from BULK.

trx_undo_report_row_operation(): Add debug assertions.

Note: Some calls to end_bulk_insert() may be redundant, but statement
boundaries are not always clear in the API (especially in the
presence of LOCK TABLES or stored procedures).

dict_table_t::n_lock_x_or_s: Keep track of LOCK_S or LOCK_X on the table.

lock_table_other_has_incompatible(): In the likely case that no
transaction is waiting for or holding LOCK_S or LOCK_X on the table,
return early: conflicts cannot exist.

This is based on the idea of Zhai Weixiang, who reported MySQL Bug #72948.

lock_table_has_to_wait_in_queue(), lock_table_dequeue():
Extend the optimization, inspired by
mysql/mysql-server@bb7191d6cbe47e15923143e194c03406cff9024b
by Jakub Łopuszański.

lock_wait(): Replaces lock_wait_suspend_thread(). Wait for the lock to
be granted or the transaction to be killed using mysql_cond_timedwait()
or mysql_cond_wait().

lock_wait_end(): Replaces que_thr_end_lock_wait() and
lock_wait_release_thread_if_suspended().

lock_wait_timeout_task: Remove. The operating system kernel will
resume the mysql_cond_timedwait() in lock_wait(). An added benefit
is that innodb_lock_wait_timeout no longer has a 'jitter' of 1 second,
which was caused by this wake-up task waking up only once per second,
and then waking up any threads for which the timeout (which was only
measured in seconds) was exceeded.

innobase_kill_query(): Set trx->error_state=DB_INTERRUPTED,
so that a call trx_is_interrupted(trx) in lock_wait() can be avoided.

We will protect things more consistently with lock_sys.wait_mutex,
which will be moved below lock_sys.mutex in the latching order.

trx_lock_t::cond: Condition variable for !wait_lock, used with
lock_sys.wait_mutex.

srv_slot_t: Remove. Replaced by trx_lock_t::cond,

lock_grant_after_reset(): Merged to to lock_grant().

lock_rec_get_index_name(): Remove.

lock_sys_t: Introduce wait_pending, wait_count, wait_time, wait_time_max
that are protected by wait_mutex.

trx_lock_t::que_state: Remove.

que_thr_state_t: Remove QUE_THR_COMMAND_WAIT, QUE_THR_LOCK_WAIT.

que_thr_t: Remove is_active, start_running(), stop_no_error().

que_fork_t::n_active_thrs, trx_lock_t::n_active_thrs: Remove.

que_thr_t::fork_type: Remove.

QUE_THR_SUSPENDED, TRX_QUE_COMMITTING: Remove.

Cleanup lock_cancel_waiting_and_release()

LOCK_MAX_N_STEPS_IN_DEADLOCK_CHECK, LOCK_MAX_DEPTH_IN_DEADLOCK_CHECK,
LOCK_RELEASE_INTERVAL: Replace with the bare use of the constants.

lock_rec_create_low(): Remove LOCK_PAGE_BITMAP_MARGIN altogether.
We already have REDZONE_SIZE as a 'safety margin' in AddressSanitizer
builds, to catch any out-of-bounds access.

lock_prdt_add_to_queue(): Avoid a useless search when enqueueing
a waiting lock request.

lock_prdt_lock(): Reduce the size of the trx->mutex critical section.

Any transaction that has requested a lock must have trx->id!=0.

trx_print_low(): Distinguish non-locking or inactive transaction
objects by displaying the pointer in parentheses.

fill_trx_row(): Do not try to map trx->id to a pointer-based value.

Support membership tests in SSPI with special prefix form

CREATE USER u IDENTIFIED WITH gssapi AS "GROUP:<group_name>"
or
CREATE USER u IDENTIFIED WITH gssapi AS "SID:<sid>"

If user is created as one of the above, after successful SSPI handshake,
this will happen

1) If "GROUP:" prefix is used, then <group_name> is translated to SID
using LookupAccountName() API

2) SSPI user is checked for  SID membership with
ImpersonateSecurityContext() and CheckMembership() APIs

Note, that it <group>/<sid> do not need strictly to refer to an actual
group.
Identity test is also supported, e.g  "GROUP:<users_name>" or
"SID:<user_sid>" will work too.


Well-known SIDs (in SDDL syntax) appear to be supported such as
"SID:WD" will refer to World/Everyone (== "SID:S-1-1-0")
or
"SID:BA" will refer to Administrators (== "SID:S-1-5-32-544")

In UAC environments, for successful checks against Administrators group,
elevation(Run As Administrator) might be necessary, since CheckMembership()
needs groups to be marked as enabled in the token group list.

There are no IO threads anymore.

* galera
* galera_sr
* galera_3nodes

Correct test.

the top level