MDEV-25334 FTWRL/Backup blocks DDL on temporary tables with binlog enabled, assertion fails in Diagnostics_area::set_error_status

Fixed by adding a MDL_BACKUP_COMMIT lock before altering temporary tables
whose creation was logged to binary log (in which case the ALTER TABLE
must also be logged)

Problem:

The problem happened because of a conceptual flaw in the server code:

a. The table level CHARSET/COLLATE clause affected all data types,
  including numeric and temporal ones:

   CREATE TABLE t1 (a INT) CHARACTER SET utf8 [COLLATE utf8_general_ci];

  In the above example, the Column_definition_attributes
  (and then the FRM record) for the column "a" erroneously inherited
  "utf8" as its character set.

b. The "ALTER TABLE t1 CONVERT TO CHARACTER SET csname" statement
   also erroneously affected Column_definition_attributes::charset
   for numeric and temporal data types and wrote "csname" as their
   character set into FRM files.

So now we have arbitrary non-relevant charset ID values for numeric
and temporal data types in all FRM files in the world :)

The code in the server and the other engines did not seem to be affected
by this flaw. Only InnoDB inplace ALTER was affected.

Solution:

Fixing the code in the way that only character string data types
(CHAR,VARCHAR,TEXT,ENUM,SET):
- inherit the table level CHARSET/COLLATE clause
- get the charset value according to "CONVERT TO CHARACTER SET csname".

Numeric and temporal data types now always get &my_charset_numeric
in Column_definition_attributes::charset and always write its ID into FRM files:
- no matter what the table level CHARSET/COLLATE clause is, and
- no matter what "CONVERT TO CHARACTER SET" says.

Details:

1. Adding helper classes to pass small parts of HA_CREATE_INFO
   into Type_handler methods:

   - Column_derived_attributes - to pass table level CHARSET/COLLATE,
     so columns that do not have explicit CHARSET/COLLATE clauses
     can derive them from the table level, e.g.

       CREATE TABLE t1 (a VARCHAR(1), b CHAR(1)) CHARACTER SET utf8;

   - Column_bulk_alter_attributes - to pass bulk attribute changes
     generated by the ALTER related code. These bulk changes affect
     multiple columns at the same time:

       ALTER TABLE ... CONVERT TO CHARACTER SET csname;

   Note, passing the whole HA_CREATE_INFO directly to Type_handler
   would not be good: HA_CREATE_INFO is huge and would need not desired
   dependencies in sql_type.h and sql_type.cc. The Type_handler API should
   use smallest possible data types!

2. Type_handler::Column_definition_prepare_stage1() is now responsible
   to set Column_definition::charset properly, according to the data type,
   for example:

   - For string data types, Column_definition_attributes::charset is set from
     the table level CHARSET/COLLATE clause (if not specified explicitly in
     the column definition).

   - For numeric and temporal fields, Column_definition_attributes::charset is
     set to &my_charset_numeric, no matter what the table level
     CHARSET/COLLATE says.

   - For GEOMETRY, Column_definition_attributes::charset is set to
     &my_charset_bin, no matter what the table level CHARSET/COLLATE says.

   Previously this code (setting `charset`) was outside of of
   Column_definition_prepare_stage1(), namely in
   mysql_prepare_create_table(), and was erroneously called for
   all data types.

3. Adding Type_handler::Column_definition_bulk_alter(), to handle
   "ALTER TABLE .. CONVERT TO". Previously this code was inside
   get_sql_field_charset() and was erroneously called for all data types.

4. Removing the Schema_specification_st parameter from
   Type_handler::Column_definition_redefine_stage1().
   Column_definition_attributes::charset is now fully properly initialized by
   Column_definition_prepare_stage1(). So we don't need access to the
   table level CHARSET/COLLATE clause in Column_definition_redefine_stage1()
   any more.

5. Other changes:
   - Removing global function get_sql_field_charset()

   - Moving the part of the former get_sql_field_charset(), which was
     responsible to inherit the table level CHARSET/COLLATE clause to
     new methods:
      -- Column_definition_attributes::explicit_or_derived_charset() and
      -- Column_definition::prepare_charset_for_string().
     This code is only needed for string data types.
     Previously it was erroneously called for all data types.

   - Moving another part, which was responsible to apply the
     "CONVERT TO" clause, to
     Type_handler_general_purpose_string::Column_definition_bulk_alter().

   - Replacing the call for get_sql_field_charset() in sql_partition.cc
     to sql_field->explicit_or_derived_charset() - it is perfectly enough.
     The old code was redundant: get_sql_field_charset() was called from
     sql_partition.cc only when there were no a "CONVERT TO CHARACTER SET"
     clause involved, so its purpose was only to inherit the table
     level CHARSET/COLLATE clause.

   - Moving the code handling the BINCMP_FLAG flag from
     mysql_prepare_create_table() to
     Column_definition::prepare_charset_for_string():
     This code is responsible to resolve the BINARY comparison style
     into the corresponding _bin collation, to do the following transparent
     rewrite:
        CREATE TABLE t1 (a VARCHAR(10) BINARY) CHARSET utf8;  ->
        CREATE TABLE t1 (a VARCHAR(10) CHARACTER SET utf8 COLLATE utf8_bin);
     This code is only needed for string data types.
     Previously it was erroneously called for all data types.

6. Renaming Table_scope_and_contents_source_pod_st::table_charset
   to alter_table_convert_to_charset, because the only purpose it's used for
   is handlering "ALTER .. CONVERT". The new name is much more self-descriptive.

SI_USER is, however in FreeBSD there are a couple of non-kernel
user signal infomations above SI_KERNEL.

Put a fallback just in case there is nothing available.

Added handling for sql_safe_updated i.e. we disable it while
we do wsrep_schema operations.

`WSREP_CLIENT` is used as condition for starting ALTER/OPTIMIZE/REPAIR TOI.
Using this condition async replicated affected DDL's will not be replicated.
Fixed by removing this condition.
Reviewed-by: Jan Lindström <jan.lindstrom@mariadb.com>

This patch makes the following changes around variable wsrep_on:

1) Variable wsrep_on can no longer be updated from a session that has
an active transaction running. The original behavior allowed cases
like this:

     BEGIN;
     INSERT INTO t1 VALUES (1);
     SET SESSION wsrep_on = OFF;
     INSERT INTO t1 VALUES (2);
     COMMIT;

With regular transactions this would result in no replication
events (not even value 1). With streaming replication it would be
unnecessarily complex to achieve the same behavior. In the above
example, it would be possible for value 1 to be already replicated if
it happened to fill a separate fragment, while value 2 wouldn't.

2) Global variable wsrep_on no longer affects current sessions, only
subsequent ones. This is to avoid a similar case to the above, just
using just by using global wsrep_on instead session wsrep_on:

      --connection conn_1
      BEGIN;
      INSERT INTO t1 VALUES(1);

      --connection conn_2
      SET GLOBAL wsrep_on = OFF;

      --connection conn_1
      INSERT INTO t1 VALUES(2);
      COMMIT;

The above example results in the transaction to be replicated, as
global wsrep_on will only affect the session wsrep_on of new
connections.
Reviewed-by: Jan Lindström <jan.lindstrom@mariadb.com>

MDEV-17913 Encrypted transactional Aria tables remain corrupt after crash recovery, automatic repairment does not work

This was because of a wrong test in encryption code that wrote random
numbers over the LSN for pages for transactional Aria tables during repair.
The effect was that after an ALTER TABLE ENABLE KEYS of a encrypted
recovery of the tables would not work.

The test cases will be pushed into 10.5 as it requires of several changes
to check table that safer not to backport.

- Aborting of fulltext index creation fails to remove the
index from sys indexes table. When we try to reload the
table definition, InnoDB fails with index count mismatch
error. InnoDB should remove the index from sys indexes while
rollbacking the secondary index creation.

- Post push to address 32-bit build failure.

- Importing table operation fails to punch the hole in
the filesystem when page compressed table is involved.
To achieve that, InnoDB firstly punches the hole for
the IOBuffer size(1MB). After that, InnoDB should write
page by page when page compression is involved.

Add condition on trx->state == TRX_STATE_COMMITTED_IN_MEMORY in order to
avoid unnecessary work. If a transaction has already been committed or
rolled back, it will release its locks in lock_release() and let
the waiting thread(s) continue execution.

Let BF wait on lock_rec_has_to_wait and if necessary other BF
is replayed.

wsrep_trx_order_before
  If BF is not even replicated yet then they are ordered
  correctly.

bg_wsrep_kill_trx
  Make sure victim_trx is found and check also its state. If
  state is TRX_STATE_COMMITTED_IN_MEMORY transaction is
  already committed or rolled back and will release it locks
  soon.

wsrep_assert_no_bf_bf_wait
  Transaction requesting new record lock should be TRX_STATE_ACTIVE
  Conflicting transaction can be in states TRX_STATE_ACTIVE,
  TRX_STATE_COMMITTED_IN_MEMORY or in TRX_STATE_PREPARED.
  If conflicting transaction is already committed in memory or
  prepared we should wait. When transaction is committed in memory we
  held trx mutex, but not lock_sys->mutex. Therefore, we
  could end here before transaction has time to do lock_release()
  that is protected with lock_sys->mutex.

lock_rec_has_to_wait
  We very well can let bf to wait normally as other BF will be
  replayed in case of conflict. For debug builds we will do
  additional sanity checks to catch unsupported bf wait if any.

wsrep_kill_victim
  Check is victim already in TRX_STATE_COMMITTED_IN_MEMORY state and
  if it is we can return.

lock_rec_dequeue_from_page
lock_rec_unlock
  Remove unnecessary wsrep_assert_no_bf_bf_wait function calls.
  We can very well let BF wait here.

Amends 48141f3c

This script is unused and unmaintained.

The logic is implemented in scripts/mysql_system_tables_fix.sql that forms part of mysql_upgrade

Its components:

  alter table mysql.user drop column `password_last_changed`, drop column `password_lifetime`, drop column `account_locked`;

has a friendlier migration path coming MDEV-24122

  alter table mysql.user change column `authentication_string` `auth_string` text COLLATE utf8_bin NOT NULL;

Already part of scripts/mysql_system_tables_fix.sql

  alter table mysql.user add column  `Password` char(41) CHARACTER SET latin1 COLLATE latin1_bin NOT NULL DEFAULT '' after `user`, add column  `is_role` enum('N','Y') CHARACTER SET utf8 NOT NULL DEFAULT 'N' after `auth_string`;

  alter table mysql.user add column `default_role` char(80) COLLATE utf8_bin NOT NULL DEFAULT '', add column `max_statement_time` decimal(12,6) NOT NULL DEFAULT '0.000000';

corrected in MDEV-23201 to be in the right order.

  update mysql.user set `password`=`auth_string`, plugin='' where plugin="mysql_native_password";

Is handled in server in the function acl_load.

Signed-off-by: Julius Goryavsky <julius.goryavsky@mariadb.com>

CREATE/DROP PROCEDIRE/FUNCTION is now treated as DDL.

we need to stop server instance on upgrade, but it may be started either by SysV init script or by SystemD.
this commit adds `mysql` target to `systemctl stop` call.
`mysql` may be the name of initscript or an alias while `mariadb` is
a systemd unit file.

- jump to label ‘handle_errors’ can enter to the scope of non-POD
  ‘Geometry_buffer buffer2’

CREATE/DROP PROCEDIRE/FUNCTION is now treated as DDL.

- Invalid function name during ER_WRONG_VALUE_FOR_TYPE and ER_GIS_INVALID_DATA

- Cherry-pick 51e48b9f - vscode gitignore
- Thanks Robin Dupret for the review.

Reviewed by:daniel@mariadb.org
            holyfoot@mariadb.com

`mallinfo` is deprecated since glibc 2.33 and has been replaced by mallinfo2.
The deprecation causes building the server to fail if glibc version is > 2.33.

Check if mallinfo2 exist on the system and use it instead.

Use like this: cmake -DWITH_ASAN=ON -DWITH_ASAN_SCOPE=ON

The function row_upd_clust_step() is invoking several static functions,
some of which used to commit the mini-transaction in some cases.
If innobase_get_computed_value() would fail due to some reason,
we would fail to invoke mtr_t::commit() and release buffer pool
page latches. This would likely lead to a hanging server later.

This regression was introduced in
commit 97db6c15 (MDEV-20618).

row_upd_index_is_referenced(), row_upd_sec_index_entry(),
row_upd_sec_index_entry(): Cleanup: Replace some ibool with bool.

row_upd_clust_rec_by_insert(), row_upd_clust_rec(): Guarantee that
the mini-transaction will always remain in active state.

row_upd_del_mark_clust_rec(): Guarantee that
the mini-transaction will always remain in active state.
This fixes one "leak" of mini-transaction on DB_COMPUTE_VALUE_FAILED.

row_upd_clust_step(): Use only one return path, which will always
invoke mtr.commit(). After a failed row_upd_store_row() call, we
will no longer "leak" the mini-transaction.

This fix was verified by RQG on 10.6 (depending on MDEV-371 that
was introduced in 10.4). Unfortunately, it is challenging to
create a regression test for this, and a test case could soon become
invalid as more bugs in virtual column evaluation are fixed.

MDEV-21697: Galera assertion !wsrep_has_changes(thd) || (thd->lex->sql_command == SQLCOM_CREATE_TABLE && !thd->is_current_stmt_binlog_format_row())

Prevent adding WSREP keys with CTAS when table is is not InnoDB.
Reviewed-by: Jan Lindström <jan.lindstrom@mariadb.com>

MDEV-24954 : 10.5.9 crashes on int wsrep::client_state::ordered_commit(): Assertion `owning_thread_id_ == wsrep::this_thread::get_id()' failed.

Binlog group commit could lead to a situation where group commit leader
accesses participant thd's wsrep client state concurrently with the
thread executing the participant thd.

This is because of race condition in
MYSQL_BIN_LOG::write_transaction_to_binlog_events(),
and was fixed by moving wsrep_ordered_commit() to happen in
MYSQL_BIN_LOG::queue_for_group_commit() under protection of
LOCK_prepare_ordered mutex.

use check_grant(..., number_of_tables=1, ...) if you only need
to check privileges for one table

Thanks to Daniel Black for reporting.

           splittable derived

If one of joined tables of the processed query is a materialized derived
table (or view or CTE) with GROUP BY clause then under some conditions it
can be subject to split optimization. With this optimization new equalities
are injected into the WHERE condition of the SELECT that specifies this
derived table. The injected equalities are generated for all join orders
with which the split optimization can employed. After the best join order
has been chosen only certain of this equalities are really needed. The
others can be safely removed. If it's not done and some of injected
equalities involve expressions over semi-joins with look-up access then
the query may return a wrong result set.
This patch effectively removes equalities injected for split optimization
that are needed only at the optimization stage and not needed for execution.

Approved by serg@mariadb.com