This can cause crashes when accessing already released memory

The issue was the Item_default created a internal field, pointing to
share->default_values, to be used with the DEFAULT() function.
This does not work for BLOB fields as these are freed at end of query.
Fixed by storing BLOB field data inside and area allocated by
Item_default_value,  like we do for nondeterministic default values.

           of two table value costructors

This bug affected queries with a [NOT] IN/ANY/ALL subquery whose top level
unit contained several table value constructors.
The problem appeared because the code of the function
Item_subselect::fix_fields() that was responsible for wrapping table
value constructors encountered at the top level unit of a [NOT] IN/ANY/ALL
subquery did not take into account that the chain of the select objects
comprising the unit were not immutable.

Approved by Oleksandr Byelkin <sanja@mariadb.com>

innodb_shutdown(): Check that fil_system.temp_space is not null
before invoking a member function.

This regression was caused by the merge
commit fa1aef39
of MDEV-24340 (commit 1eb59c30).

This patch actually fixes the bug MDEV-24675 and the bug MDEV-24618:
Assertion failure when TVC uses a row in the context expecting scalar value

The cause of these bugs is the same wrong call of the function that fixes
value expressions in the value list of a table value constructor.
The assertion failure happened when an expression in the value list is of
the row type. In this case an error message was expected, but it was not
issued because the function fix_fields_if_needed() was called for to
check fields of value expressions in a TVC instead of the function
fix_fields_if_needed_for_scalar() that would also check that the value
expressions are are of a scalar type.
The first bug happened when a table value expression used an expression
returned by single-row subselect. In this case the call of the
fix_fields_if_needed_for_scalar virtual function must be provided with
and address to which the single-row subselect has to be attached.

Test cases were added for each of the bugs.

Approved by Oleksandr Byelkin <sanja@mariadb.com>

The assertion failed in handler::ha_reset upon SELECT under
READ UNCOMMITTED from table with index on virtual column.

This was the debug-only failure, though the problem is mush wider:
* MY_BITMAP is a structure containing my_bitmap_map, the latter is a raw
 bitmap.
* read_set, write_set and vcol_set of TABLE are the pointers to MY_BITMAP
* The rest of MY_BITMAPs are stored in TABLE and TABLE_SHARE
* The pointers to the stored MY_BITMAPs, like orig_read_set etc, and
 sometimes all_set and tmp_set, are assigned to the pointers.
* Sometimes tmp_use_all_columns is used to substitute the raw bitmap
 directly with all_set.bitmap
* Sometimes even bitmaps are directly modified, like in
TABLE::update_virtual_field(): bitmap_clear_all(&tmp_set) is called.

The last three bullets in the list, when used together (which is mostly
always) make the program flow cumbersome and impossible to follow,
notwithstanding the errors they cause, like this MDEV-17556, where tmp_set
pointer was assigned to read_set, write_set and vcol_set, then its bitmap
was substituted with all_set.bitmap by dbug_tmp_use_all_columns() call,
and then bitmap_clear_all(&tmp_set) was applied to all this.

To untangle this knot, the rule should be applied:
* Never substitute bitmaps! This patch is about this.
 orig_*, all_set bitmaps are never substituted already.

This patch changes the following function prototypes:
* tmp_use_all_columns, dbug_tmp_use_all_columns
 to accept MY_BITMAP** and to return MY_BITMAP * instead of my_bitmap_map*
* tmp_restore_column_map, dbug_tmp_restore_column_maps to accept
 MY_BITMAP* instead of my_bitmap_map*

These functions now will substitute read_set/write_set/vcol_set directly,
and won't touch underlying bitmaps.

Dump sequences first.

This atch made to keep it small and
to keep number of queries to the server the same.

Order of tables in a dump can not be changed
(except sequences first) because mysql_list_tables
uses SHOW TABLES and I used SHOW FULL TABLES.

We may end up with an empty leaf page (containing only an ADD COLUMN
metadata record) that is not the root page.

innobase_add_instant_try(): Disable an optimization for a non-canonical
empty table that contains a metadata record somewhere else than in
the root page.

btr_pcur_store_position(): Tolerate a non-canonical empty table.

Ever since commit 947efe17
InnoDB no longer writes binlog position in one place.
It will not at all be written to the TRX_SYS page, and
instead it will be written to the undo log header page that
changes the transaction state.

trx_rseg_mem_restore(): Recover the information from the latest
written page.

I_S tables were materialized too late, an attempt to use table
statistics before the table was created caused a crash.

Let's move table creation up. it only needs read_set to
be calculated properly, this happens in JOIN::optimize_inner(),
after semijoin transformation.

Note that tables are not populated at that point, so most of the
statistics would make no sense anyway. But at least field sizes
will be correct. And it won't crash.

Fix for MDEV-23033 fixes a problem in replication applying of transactions, which contain cascading foreign key delete for a table, which has indexed virtual column.
This fix adds slave_fk_event_map flag for table, to mark when the prelocking is needed for applying of a transaction.
See commit 608b0ee5 for more details.
However, this fix is targeted for async replication only, Rows_log_event::do_apply_event() has condition to rule out galera replication from the fix domain, and use cases suffering from MDEV-23033 and related MDEV-21153 will fail in galera cluster.

The fix in this commit removes the condition to rule out the setting of slave_fk_event_map flag from galera replication, and makes the fix in MDEV-23033 effective for galera replication as well.

Finally, a mtr test for virtual column support has been added. galera.galera_virtual_column.test has as first test a scenario from MDEV-21153
Reviewed-by: Jan Lindström <jan.lindstrom@mariadb.com>

MDEV-23666: Assertion `m_cpp_buf <= ptr && ptr <= m_cpp_buf + m_buf_length' failed in Lex_input_stream::body_utf8_append

On parsing statements for which a starting backtick (`) delimiter doesn't have
a corresponding ending backtick, a current pointer to a position inside a
pre-processed buffer could go beyond the end of the buffer.

This bug report caused by the commit d4967659
  "MDEV-22022 Various mangled SQL statements will crash 10.3 to 10.5 debug builds".

In order to fix the issue both pointers m_ptr and m_cpp_ptr must be
rolled back to previous position in raw input and pre-processed input streams
correspondingly in case end of query reached during parsing.

constellation

Analysis: The decimals is set to NOT_FIXED_DEC for Field_str even if it is
NULL. Unsigned has decimals=0. So Type_std_attributes::decimals is set to 39
(maximum between 0 and 39). This results in incorrect number of decimals
when we have union of unsigned and NULL type.

Fix: Check if the field is created from NULL value. If yes, set decimals to 0
otherwise set it to NOT_FIXED_DEC.

MDEV-17227 Server crash in TABLE_SHARE::init_from_sql_statement_string upon table discovery with non-existent database

* failed init_from_binary_frm_image can clear share->db_plugin,
  don't use it on the error path
* cleanup the test a bit

force alter_algorithm=DEFAULT in mysql_system_tables_fix.sql,
in case my.cnf sets it to something incompatible

MDEV-20763 Table corruption or Assertion `btr_validate_index(index, 0, false)' failed in row_upd_sec_index_entry with virtual column and EMPTY_STRING_IS_NULL SQL mode

unset empty_string_is_null mode when parsing generated columns in a table,
this mode affects pasring.

the test was doing

--replace_result $con_id con_id
eval SHOW EXPLAIN FOR $con_id;

with the intention of replacing the variable part of the statement
in the result log. But actually replace_result replaces everything
that matches. In particular, when $con_id is 100, the warning

Note   1003    select sleep(100000)

becomes

Note   con_id3    select sleep(con_id000)

MDEV-24524 Assertion `ls->length < 0xFFFFFFFFL && ((ls->length == 0 && !ls->str) || ls->length == strlen(ls->str))' failed in String::append on SELECT from I_S

don't expect return type of a stored function to be valid.
it's read from a table, so can be messed with.
it even can contain \0 bytes in the middle of the type name

MDEV-20502 Queries against spider tables return wrong values for columns following constant declarations.

Add test cases.

MDEV-20502 Queries against spider tables return wrong values for columns following constant declarations.

When executing a query like "select id, 0 as const, val from ...", there are 3 columns(items) in Query->select at handlerton->create_group_by(). After that, MariaDB makes a temporary table with 2 columns. The skipped items are const item, so fixing Spider to skip const items for items at Query->select.

		with wrong data type is added

  Inplace alter fails to report error when fts_doc_id column with
wrong data type is added.

prepare_inplace_alter_table_dict(): Should check whether the column
is fts_doc_id. It should be of bigint type, should accept non null
data type and it should be in capital letters.

The fix of MDEV-17556 (commit e25623e7
and commit 61a362c9) has been
omitted due to conflicts and will have to be applied separately later.

A race condition may occur between the execution of transaction commit,
and an execution of a KILL statement that would attempt to abort that
transaction.

MDEV-17092 worked around this race condition by modifying InnoDB code.
After that issue was closed, Sergey Vojtovich pointed out that this
race condition would better be fixed above the storage engine layer:

If you look carefully into the above, you can conclude that
thd->free_connection() can be called concurrently with
KILL/thd->awake(). Which is the bug. And it is partially fixed in
THD::~THD(), that is destructor waits for KILL completion:

Fix: Add necessary mutex operations to THD::free_connection()
and move WSREP specific code also there. This ensures that no
one is using THD while we do free_connection(). These mutexes
will also ensures that there can't be concurrent KILL/THD::awake().

innobase_kill_query
  We can now remove usage of trx_sys_mutex introduced on MDEV-17092.

trx_t::free()
  Poison trx->state and trx->mysql_thd

This patch is validated with an RQG run similar to the one that
reproduced MDEV-17092.

The assertion failed in handler::ha_reset upon SELECT under
READ UNCOMMITTED from table with index on virtual column.

This was the debug-only failure, though the problem is mush wider:
* MY_BITMAP is a structure containing my_bitmap_map, the latter is a raw
 bitmap.
* read_set, write_set and vcol_set of TABLE are the pointers to MY_BITMAP
* The rest of MY_BITMAPs are stored in TABLE and TABLE_SHARE
* The pointers to the stored MY_BITMAPs, like orig_read_set etc, and
 sometimes all_set and tmp_set, are assigned to the pointers.
* Sometimes tmp_use_all_columns is used to substitute the raw bitmap
 directly with all_set.bitmap
* Sometimes even bitmaps are directly modified, like in
TABLE::update_virtual_field(): bitmap_clear_all(&tmp_set) is called.

The last three bullets in the list, when used together (which is mostly
always) make the program flow cumbersome and impossible to follow,
notwithstanding the errors they cause, like this MDEV-17556, where tmp_set
pointer was assigned to read_set, write_set and vcol_set, then its bitmap
was substituted with all_set.bitmap by dbug_tmp_use_all_columns() call,
and then bitmap_clear_all(&tmp_set) was applied to all this.

To untangle this knot, the rule should be applied:
* Never substitute bitmaps! This patch is about this.
 orig_*, all_set bitmaps are never substituted already.

This patch changes the following function prototypes:
* tmp_use_all_columns, dbug_tmp_use_all_columns
 to accept MY_BITMAP** and to return MY_BITMAP * instead of my_bitmap_map*
* tmp_restore_column_map, dbug_tmp_restore_column_maps to accept
 MY_BITMAP* instead of my_bitmap_map*

These functions now will substitute read_set/write_set/vcol_set directly,
and won't touch underlying bitmaps.

MDEV-16272 rpl.rpl_semisync_ali_issues failed in buildbot, SHOW variable was done instead of waiting for the value of that variable

Use the same variable in both scopes (from where we have "goto error" and target of the goto)

Problem: Assertion `transactional_table || !changed ||
thd->transaction.stmt.modified_non_trans_table' failed due REPLACE into a
versioned table.

It is not specific to system versioning/pertitioning/heap, but this
combination makes it much easier to reproduce.

The thing is to make first ha_update_row call succeed to make
info->deleted != 0. And then make REPLACE fail by any reason.

In this scenario we overflow versioned partition, so next ha_update_row
succeeds, but corresponding ha_write_row fails to insert history record.

Fix: modified_non_trans_table is set in one missed place

fixing windows build.

mysql_col_offset was not updated after the new column has been added by an
INSTANT ALTER TABLE -- table data dictionary had been remaining the same.

When the virtual column is added or removed, table was usually evicted and
then reopened, which triggered vcol info rebuild on the next open.

However this also should be done when the usual column is added or removed:
mariadb always stores virtual field at the end of maria record,
so the shift should always happen.

Fix:
expand the eviction condition to the case when usual fields are
added/removed

Note:
this should happen only in the case of !new_clustered:
* When new_clustered is true, a new data dictionary is created, and vcol
metadata is rebuilt in `alter_rebuild_apply_log()`
* We can't do it in `new_clustered` case, because the old table is not yet
subctituted correctly