log_copying_thread(): Keep copying redo log until the end has been
reached. (Previously, we would stop copying as soon as
the first batch of xtrabackup_copy_logfile() returned.)

log_copying: Remove. Use log_copying_running instead.

copy_logfile: Remove. Log copying will now only be invoked from
2 places: from xtrabackup_backup_func() for the initial batch,
and from log_copying_thread() until all of the log has been read.
Use the global variable metadata_to_lsn for determining if the
final part of the log is being copied.

xtrabackup_copy_log(): Add diagnostic messages for terminating
the copying. These messages should be dead code, because
log_group_read_log_seg() should be checking for the same.

xtrabackup_copy_logfile(): Correct the retrying logic.
If anything was successfully read, process the portion that
was read. On failure, let the caller close dst_log_file.

io_watching_thread(): Stop throttling during the last phase
of copying the log (metadata_to_lsn!=0). The final copying
of the log will now be performed in log_copying_thread().

stop_backup_threads(): Clean up the message about stopping
the log copying thread.

xtrabackup_backup_low(): Read metadata_to_lsn from the latest
checkpoint header page, even if it is the first page.
Let the log_copying_thread take care of copying all of
the redo log.

InnoDB insisted on closing the file handle before renaming a file.
Renaming a file should never be a problem on POSIX systems. Also on
Windows it should work if the file was opened in FILE_SHARE_DELETE
mode.

fil_space_t::stop_ios: Remove. We no longer need to stop file access
during rename operations.

fil_mutex_enter_and_prepare_for_io(): Remove the wait for stop_ios.

fil_rename_tablespace(): Remove the retry logic; do not close the
file handle. Remove the unused fault injection that was added along
with the DATA DIRECTORY functionality (MySQL WL#5980).

os_file_create_simple_func(), os_file_create_func(),
os_file_create_simple_no_error_handling_func(): Include FILE_SHARE_DELETE
in the share_mode. (We will still prevent multiple InnoDB instances
from using the same files by not setting FILE_SHARE_WRITE.)

Revert part of 64094e12 because mysqld_safe isn't just used for
Debian. As such references to specific packaging files like
/etc/mysql/debian.cnf shouldn't have passed a review.

There's also no history of why 64094e12 was needed based on
the history of debian/patches/38_scripts__mysqld_safe.sh__signals.dpatch

Other distos have survived without mysqld_safe handling signals.

c902d5a4 changed this to bash.
This isn't approprate as mysqld_safe is used by FreeBSD where
there is no bash by default.

The original reason to use bash seems related to MDEV-3279 which
was a dash bug fixed in 2009 - https://git.kernel.org/pub/scm/utils/dash/dash.git/commit/?id=3800d4934391b144fd261a7957aea72ced7d47ea

Working around 9 year old fixed bugs shouldn't be done here.

ha_innobase::optimize(): If both innodb_defragment and
innodb_optimize_fulltext_only are at their default settings (OFF),
fall back to ALTER TABLE. Else process one or both options.

ha_innobase::set_partition_owner_stats(): Remove (unused function).

ha_innobase::ha_partition_stats: Remove (the variable is never read).

Remove unused ut_timer functions.

The problem described in the bug report happened because the code
did not test check_cols(1) after fix_fields() in a few places.

Additionally, fix_fields() could be called multiple times for SP variables,
because they are all fixed at a early stage in append_for_log().

Solution:
1. Adding a few helper methods
   - fix_fields_if_needed()
   - fix_fields_if_needed_for_scalar()
   - fix_fields_if_needed_for_bool()
   - fix_fields_if_needed_for_order_by()
  and using it in many cases instead of fix_fields() where
  the "fixed" status is not definitely known to be "false".

2. Adding DBUG_ASSERT(!fixed) into Item_splocal*::fix_fields()
   to catch double execution.

3. Adding tests.

As a good side effect, the patch removes a lot of duplicate code (~60 lines):

   if (!item->fixed &&
       item->fix_fields(..) &&
       item->check_cols(1))
     return true;

After a failed ADD INDEX, dict_index_remove_from_cache_low()
could iterate the index fields and dereference a freed virtual
column object when trying to remove the index from the v_indexes
of the virtual column.

This regression was caused by a merge of
MDEV-16119 InnoDB lock->index refers to a freed object.

ha_innobase_inplace_ctx::clear_added_indexes(): Detach the
indexes of uncommitted indexes from virtual columns, so that
the iteration in dict_index_remove_from_cache_low() can be avoided.

ha_innobase::prepare_inplace_alter_table(): Ignore uncommitted
corrupted indexes when rejecting ALTER TABLE. (This minor bug was
revealed by the extension of the test case.)

dict_index_t::detach_columns(): Detach an index from virtual columns.
Invoked by both dict_index_remove_from_cache_low() and
ha_innobase_inplace_ctx::clear_added_indexes().

dict_col_t::detach(const dict_index_t& index): Detach an index from
a column.

dict_col_t::is_virtual(): Replaces dict_col_is_virtual().

dict_index_t::has_virtual(): Replaces dict_index_has_virtual().

log_crypt_101_read_block(): Mimic MariaDB 10.1, and use the first
encryption key if the key for the checkpoint cannot be found.
Redo log encryption key rotation was ultimately disabled
in MDEV-9422 (MariaDB 10.1.13) due to design issues. So, from
MariaDB 10.1.13 onwards only one log encryption key should matter.

recv_log_format_0_recover(): Add the parameter 'bool crypt'.
Indicate when the log cannot be decrypted for upgrade, instead of
making a possibly false claim that the log requires crash recovery.

init_crypt_key(): Remove extra space from a message.

just before they're sent to the client. Because mysql->host
value is known only after the connection is established,
and it's needed for the "_sever_host" attribute.

This fixes ASAN use-after-free warning in rpl tests

followup for ee8dfc68

MDEV-15991 Server crashes in setup_on_expr upon calling SP or function executing DML on versioned tables

Do not try to set versioning conditions on every SP call. It may work
incorrectly, but it's a general bug described in MDEV-774.
This patch makes system versioning stuff consistent with other code and
also fixes a use-after-free bug.

Closes #756

* Always-false vers_select_conds_t::from_query
* Debug helpers

Closes #684

This partially reverts commit dc0a7660.

Make mariadb crc32 lib platform independent

It looks strange that someone can make use of 2 crc libraries
(Power64 or AArch64) at the same time.

The patch sets macros 'CRC32_LIBRARY' to make platform independence as an optional crc32 library.

Change-Id: I68bbf73cafb6a12f7fb105ad57d117b114a8c4af
Signed-off-by: Yuqi Gu <yuqi.gu@arm.com>

Just spacing out the links on separate linkes, they were running together
in a block of text and could be a little hard for some people to
differentiate where one begins and the next ends. Seems silly, but just
trying to help in the formatting a bit.

This bug caused crashes for queries with unreferenced non-recursive
CTEs specified by unions.It happened because the function
st_select_lex_unit::prepare() tried to use the value of the field 'derived'
that could not be set for unferenced CTEs as there was no derived
table associated with an unreferenced CTE.

INNOBASE_DEFAULTS: Replace ALTER_ADD_COLUMN with the more appropriate
ALTER_ADD_STORED_BASE_COLUMN. This clean-up causes no change of
behaviour, because ALGORITHM=INPLACE would be refused for
ALTER_ADD_STORED_GENERATED_COLUMN, and default values are not computed
nor substituted for ALTER_ADD_VIRTUAL_COLUMN.

Disable /permissive- flag, so MFC  is found, and upgrade_wizard is built.

MFC code is not standard C++, so it doesn not play well with /permissive-

commit 2dbeebdb accidentally changed
ALTER_COLUMN_OPTION and ALTER_COLUMN_STORAGE_TYPE to be separate flags.
InnoDB and Mroonga are only checking for the latter;
the example storage engine is checking for the former only.

The impact of this bug should be incorrect operation of Mroonga when
the column options GROONGA_TYPE, FLAGS are changed.

InnoDB does not define any column options, only table options,
so the flag ALTER_COLUMN_OPTION should never have been set.

Also, remove the unused flag ALTER_DROP_HISTORICAL.

Fix type mismatches in the unit test mdev10259().

btr_search_info_get_ref_count(): Do not return early if !table->space.
We can simply access table->space_id even after the tablespace has
been discarded.

btr_get_search_latch(): Relax a debug assertion to allow
!index->table->space.

Include all the Makefiles that define variables that can be useful
within debian/rules. This includes buildflags.mk as well.

Use the standard variable names and don't define our own.

For details see https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=852728

For details see https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=865737

Also fixes MDEV-14727, MDEV-14491
InnoDB: Error: Waited for 5 secs for hash index ref_count (1) to drop to 0
by replacing the flawed wait logic in dict_index_remove_from_cache_low().

On DISCARD TABLESPACE, there is no need to drop the adaptive hash index.
We must drop it on IMPORT TABLESPACE, and eventually on DROP TABLE or
DROP INDEX. As long as the dict_index_t object remains in the cache
and the table remains inaccessible, the adaptive hash index entries
to orphaned pages would not do any harm. They would be dropped when
buffer pool pages are reused for something else.

btr_search_drop_page_hash_when_freed(), buf_LRU_drop_page_hash_batch():
Remove the parameter zip_size, and pass 0 to buf_page_get_gen().

buf_page_get_gen(): Ignore zip_size if mode==BUF_PEEK_IF_IN_POOL.

buf_LRU_drop_page_hash_for_tablespace(): Drop the adaptive hash index
even if the tablespace is inaccessible.

buf_LRU_drop_page_hash_for_tablespace(): New global function, to drop
the adaptive hash index.

buf_LRU_flush_or_remove_pages(), fil_delete_tablespace():
Remove the parameter drop_ahi.

dict_index_remove_from_cache_low(): Actively drop the adaptive hash index
if entries exist. This should prevent InnoDB hangs on DROP TABLE or
DROP INDEX.

row_import_for_mysql(): Drop any adaptive hash index entries for the table.

row_drop_table_for_mysql(): Drop any adaptive hash index for the table,
except if the table resides in the system tablespace. (DISCARD TABLESPACE
does not apply to the system tablespace, and we do no want to drop the
adaptive hash index for other tables than the one that is being dropped.)

row_truncate_table_for_mysql(): Drop any adaptive hash index entries for
the table, except if the table resides in the system tablespace.