The aim of the InnoDB change buffer is to avoid delays when a leaf page
of a secondary index is not present in the buffer pool, and a record needs
to be inserted, delete-marked, or purged. Instead of reading the page into
the buffer pool for making such a modification, we may insert a record to
the change buffer (a special index tree in the InnoDB system tablespace).
The buffered changes are guaranteed to be merged if the index page
actually needs to be read later.

The change buffer could be useful when the database is stored on a
rotational medium (hard disk) where random seeks are slower than
sequential reads or writes.

Obviously, the change buffer will cause write amplification, due to
potentially large amount of metadata that is being written to the
change buffer. We will have to write redo log records for modifying
the change buffer tree as well as the user tablespace. Furthermore,
in the user tablespace, we must maintain a change buffer bitmap page
that uses 2 bits for estimating the amount of free space in pages,
and 1 bit to specify whether buffered changes exist. This bitmap needs
to be updated on every operation, which could reduce performance.

Even if the change buffer were free of bugs such as MDEV-24449
(potentially causing the corruption of any page in the system tablespace)
or MDEV-26977 (corruption of secondary indexes due to a currently
unknown reason), it will make diagnosis of other data corruption harder.

Because of all this, it is best to disable the change buffer by default.

The problem was that "group_min_max optimization" does not work if
some aggregate functions, like COUNT(*), is used.
The function get_best_group_min_max() is using the join->sum_funcs
array to check which aggregate functions are used.
The bug was that aggregates in HAVING where not yet added to
join->sum_funcs at the time get_best_group_min_max() was called.

Fixed by populate join->sum_funcs already in prepare, which means that
all sum functions will be in join->sum_funcs in get_best_group_min_max().
A benefit of this approach is that we can remove several calls to
make_sum_func_list() from the code and simplify the function.

I removed some wrong setting of 'sort_and_group'.
This variable is set when alloc_group_fields() is called, as part
of allocating the cache needed by end_send_group() and does not need
to be set by other functions.

One problematic thing was that Spider is using *join->sum_funcs to detect
at which stage the optimizer is and do internal calculations of aggregate
functions. Updating join->sum_funcs early caused Spider to fail when trying
to find min/max values in opt_sum_query().
Fixed by temporarily resetting sum_funcs during opt_sum_query().

Reviewer: Sergei Petrunia

The problem was that get_best_group_min_max() did not check if fields used
by the "group_min_max optimization" where used in sub queries.
Because of this, it did not detect that a key (b,a) was used in the WHERE
clause for the statement:
SELECT DISTINCT b FROM t1 WHERE EXISTS ( SELECT 1 FROM DUAL WHERE a > 1 ).

Fixed by also traversing the sub queries when checking if a field is used.
This disables group_min_max_optimization for the above query.

Reviewer: Sergei Petrunia

MENT-328 wrongly assumed that the backup failed because of warnings from
mariabackup about not found files. This is normal (and the error message
should be deleted).

randgen failed because mariabackup didn't retry BACKUP STAGE BLOCK DDL
if it failed with a deadlock.

To simplify things, I implemented the retry loop in the server as
this particular deadlock should be quickly resolved.

When a server is compiled with -fPIE, my_addr_resolve needs to
subtract the info.dli_fbase from symbol addresses in memory for
addr2line to recognize them.  When a server is compiled without -fPIE,
my_addr_resolve should not do it.  Unfortunately not all compilers
define __PIE__ when -fPIE was used (e.g. older gcc doesn't), so we
have to resort to run-time detection.

If innodb_flush_method=O_DSYNC, log_sys.flushed_to_disk_lsn  is changed
without 'flush_lock' protection inside log_write().

This leads to a race condition, if there are 2 threads running in parallel,
doing log_write_up_to() with different values for 'flush_to_disk'

In this case, log_write() and log_write_flush_to_disk_low() can execute at
the same time, and both would change flushed_lsn.

The fix is to remove special treatment of durable writes from log_write().
There is no apparent reason for this special treatment, log_write_flush_to_disk_low()
is already optimized for durable writes.

Nor there is an apparent reason to call log_flush_notify() more often in
for O_DSYNC.

fixes errors on rpm-*-debug builder

when it's run directly after main.mysql_json_mysql_upgrade

because mysqld--help-aria starts a second mysqld that reads the plugin
table, so it has to be flushed and closed at that time.

this fixes plugins.qc_info in --embed

followup for 430d60d1 MDEV-24487

this fixes main.partition_cache and main.cache_innodb in --embed

followup for 430d60d1 MDEV-24487

MDEV-27668 Assertion `item->type_handler()->is_traditional_scalar_type() || item->type_handler() == type_handler()' failed in Field_inet6::can_optimize_keypart_ref

The correct macro to detect the AMD64 ISA is _M_X64, not M_IX64.

Thanks to Vladislav Vaintroub for pointing this out.

MDEV-24487 Error after update to 10.5.8 on CentOS-8: DBD::mysql::st execute failed: Unknown MySQL error

The problem happened because the the new client capability flag
CLIENT_EXTENDED_METADATA was not put into the cache entry key.
So results cached by a new client were sent to the old client (and vica versa)
with a mis-matching metadata, which made the client abort the connection on
an unexpected result set metadata packet format.

The problem was caused by the patch for:
  MDEV-17832 Protocol: extensions for Pluggable types and JSON, GEOMETRY
which forgot to adjust the query cache code.

Fix:

- Adding a new member Query_cache_query_flags::client_extended_metadata,
  so only clients with equal CLIENT_EXTENDED_METADATA flag values can
  reuse results.

- Adding a new column CLIENT_EXTENDED_METADATA into
  INFORMATION_SCHEMA.QUERY_CACHE_INFO (privided by the qc_info plugin).

MDEV-26223 Galera cluster node consider old server_id value even after modification of server_id [wsrep_gtid_mode=ON]

Variable `wsrep_new_cluster` now will be TRUE also when there is only `gcomm://` used
in configuration. This configuration, even without --wsrep-new-cluster,
is considered to bootstrap new cluster.

Updated galera GTID test to ignore warning message when non bootstrap
node have server-id different thant one cluster is initialized with.
Reviewed-by: Jan Lindström <jan.lindstrom@mariadb.com>

In recv_sys_t::apply(), we were unnecessarily looking up pages
in buf_pool.page_hash and potentially waiting for exclusive page latches.

Before buf_page_get_low() would return an x-latched page,
that page will have to be read and buf_page_read_complete() would
have invoked recv_recover_page() to apply the log to the page.

Therefore, it suffices to invoke recv_read_in_area() to trigger
a transition from RECV_NOT_PROCESSED.

recv_read_in_area(): Take the iterator as a parameter, and remove
page_id lookups. Should the page already be in buf_pool.page_hash,
buf_page_init_for_read() will return nullptr to buf_read_page_low()
and buf_read_page_background().

recv_sys_t::apply(): Replace goto, remove dead code, and add assertions
to guarantee that the iteration will make progress.

Reviewed by: Vladislav Lesin

Adjusting rocksdb tests results.

Revert "MDEV-26223 Galera cluster node consider old server_id value even after modification of server_id [wsrep_gtid_mode=ON]"

This reverts commit a0f711e9.

The 10.5 version of the patch.

Removing DEFAULT from INFORMATION_SCHEMA columns.
DEFAULT in read-only tables is rather meaningless.
Upgrade should go smoothly.

Also fixes:
 MDEV-20254 Problems with EMPTY_STRING_IS_NULL and I_S tables

Hybrid functions (IF, COALESCE, etc) did not preserve the JSON property
from their arguments. The same problem was repeatable for single row subselects.

The problem happened because the method Item::is_json_type() was inconsistently
implemented across the Item hierarchy. For example, Item_hybrid_func
and Item_singlerow_subselect did not override is_json_type().

Solution:

- Removing Item::is_json_type()

- Implementing specific JSON type handlers:
  Type_handler_string_json
  Type_handler_varchar_json
  Type_handler_tiny_blob_json
  Type_handler_blob_json
  Type_handler_medium_blob_json
  Type_handler_long_blob_json

- Reusing the existing data type infrastructure to pass JSON
  type handlers across all item types, including classes Item_hybrid_func
  and Item_singlerow_subselect. Note, these two classes themselves do not
  need any changes!

- Extending the data type infrastructure so data types can inherit
  their properties (e.g. aggregation rules) from their base data types.
  E.g. VARCHAR/JSON acts as VARCHAR, LONGTEXT/JSON acts as LONGTEXT
  when mixed to a non-JSON data type. This is done by:
    - adding virtual method Type_handler::type_handler_base()
    - adding a helper class Type_handler_pair
    - refactoring Type_handler_hybrid_field_type methods
      aggregate_for_result(), aggregate_for_min_max(),
      aggregate_for_num_op() to use Type_handler_pair.

This change also fixes:

  MDEV-27361 Hybrid functions with JSON arguments do not send format metadata

Also, adding mtr tests for JSON replication. It was not covered yet.
And the current patch changes the replication code slightly.

Problem:
=======
  InnoDB ran out of memory during recovery and it fails to
flush the dirty LRU blocks. The reason is that buffer pool
can ran out before the LRU list length reaches
BUF_LRU_OLD_MIN_LEN(256) threshold.

Fix:
====
During recovery, InnoDB should write out and evict all
dirty blocks.

MDEV-26223 Galera cluster node consider old server_id value even after modification of server_id [wsrep_gtid_mode=ON]

For non bootstrap node server id should be ignored because using custom
value can lead to inconsistency problem with replicated GTID in cluster.
Providing warning message when this happens.
Reviewed-by: Jan Lindström <jan.lindstrom@mariadb.com>

A query in form

  SELECT DISTINCT expr_that_is_inferred_to_be_const LIMIT 0 OFFSET n

produces one row when it should produce none. The issue was in
JOIN_TAB::remove_duplicates() in the piece of logic that tried to
avoid duplicate removal for such cases but didn't account for possible
"LIMIT 0".

Fixed by making Select_limit_counters::set_limit() change OFFSET to 0
when LIMIT is 0.

The code was backported from 10.6 bd03c0e5
commit. See that commit message for details.

Apart from the above commit trx_lock_t::wait_trx was also backported from
MDEV-24738. trx_lock_t::wait_trx is protected with lock_sys.wait_mutex
in 10.6, but that mutex was implemented only in MDEV-24789. As there is no
need to backport MDEV-24789 for MDEV-27025,
trx_lock_t::wait_trx is protected with the same mutexes as
trx_lock_t::wait_lock.

This fix should not break innodb-lock-schedule-algorithm=VATS. This
algorithm uses an Eldest-Transaction-First (ETF) heuristic, which prefers
older transactions over new ones. In this fix we just insert granted lock
just before the last granted lock of the same transaction, what does not
change transactions execution order.

The changes in lock_rec_create_low() should not break Galera Cluster,
there is a big "if" branch for WSREP. This branch is necessary to provide
the correct transactions execution order, and should not be changed for
the current bug fix.

In commit 4c3ad244 (MDEV-27416)
an unnecessarily strict wait condition was introduced in the
function buf_flush_wait(). Most callers actually only care that
the pages have been flushed, not that a checkpoint has completed.

Only in the buf_flush_sync() call for log resizing, we might care
about the log checkpoint. But, in fact,
srv_prepare_to_delete_redo_log_file() is explicitly disabling
checkpoints. So, we can simply remove the unnecessary wait loop.

Thanks to Krunal Bauskar for reporting this performance regression
that we failed to repeat in our testing.

bump maturity to beta

MDEV-25373 DROP TABLE doesn't raise error while dropping non-existing table in MariaDB 10.5.9 when OQGraph SE is loaded to the server

don't auto-succeed every DROP TABLE

buf_pool_t::realloc(): Invoke page_cleaner_wakeup()
if buf_LRU_get_free_only() returns a null pointer.

Ever since commit 7b1252c0 (MDEV-24278)
the page cleaner would remain in untimed sleep, expecting explicit
calls to buf_pool_t::page_cleaner_wakeup() when the ratio of dirty pages
could change.

Failure to wake up the page cleaner will cause all page writes to be
initiated by buf_flush_LRU_list_batch(). That might work too,
provided that the buffer pool size is at least BUF_LRU_MIN_LEN (256)
pages, but it would not advance the log checkpoint.

Windows builds failed due to the following error:
'#': invalid character: possibly the result of a macro expansion

The commit e954d9de gave different lifetime to wide_share and
partition_handler_share. This introduced the possibility that
partition_handler_share could be accessed even after it was freed.

We stop sharing partitoiin_handler_share and make it belong to
a single wide_handler to fix the problem.

In MDEV-14425, an early plan was to introduce a separate log file
for file-level records and checkpoint information. The reasoning was
that fil_system.mutex contention would be reduced by not having to
maintain fil_system.named_spaces. The mutex contention was actually
fixed in MDEV-23855 by making some data fields in fil_space_t and
fil_node_t use std::atomic.

Using a single circular log file simplifies recovery and backup.