on tablespace files

Fix is to always add Full Control for NetworkService account, for every
file that copyback/moveback copies around.

If required privilege is missing, dump the output from "SHOW GRANTS"
into mariabackup log.

This will help troubleshooting, and make the bug reproducible.

The statement

SET GLOBAL innodb_encryption_rotate_key_age=0;

would have the unwanted side effect that ENCRYPTION=DEFAULT tablespaces
would no longer be encrypted or decrypted according to the setting of
innodb_encrypt_tables.

We implement a trigger, so that whenever one of the following is executed:

SET GLOBAL innodb_encrypt_tables=OFF;
SET GLOBAL innodb_encrypt_tables=ON;
SET GLOBAL innodb_encrypt_tables=FORCE;

all wrong-state ENCRYPTION=DEFAULT tablespaces will be added to
fil_system_t::rotation_list, so that the encryption will be added
or removed.

Note: This will *NOT* happen automatically after a server restart.
Before reading the first page of a data file, InnoDB cannot know
the encryption status of the data file. The statement
SET GLOBAL innodb_encrypt_tables will have the side effect that
all not-yet-read InnoDB data files will be accessed in order to
determine the encryption status.

innodb_encrypt_tables_validate(): Stop disallowing
SET GLOBAL innodb_encrypt_tables when innodb_encryption_rotate_key_age=0.
This reverts part of commit 50eb40a2
that addressed MDEV-11738 and MDEV-11581.

fil_system_t::read_page0(): Trigger a call to fil_node_t::read_page0().
Refactored from fil_space_get_space().

fil_crypt_rotation_list_fill(): If innodb_encryption_rotate_key_age=0,
initialize fil_system->rotation_list. This is invoked both on
SET GLOBAL innodb_encrypt_tables and
on SET GLOBAL innodb_encryption_rotate_key_age=0.

fil_space_set_crypt_data(): Remove.

fil_parse_write_crypt_data(): Simplify the logic.

This is joint work with Marko Mäkelä.

MDEV-17654 Incorrect syntax returned for column with CHECK constraint in the "SHOW CREATE TABLE ..." result

Prepend COMMENT before CHECK constraint in SHOW CREATE

Closes #924

The working directory, resource limits and core pattern will
aid the user finding a core file in the case of failure.

While the core file size is most relevant however other resource
limits may give a clue as the the cause of the fatal signal so
include them also.

As signal handler functions are limited, proc filesystem reads/
readlink calls are used instead of the more obvious getcwd/getrlimits
functions which aren't listed as signal safe.

Results in output of the form:

Writing a core file: working directory at /tmp/datadir
Resource Limits:
Limit                     Soft Limit           Hard Limit Units
Max cpu time              unlimited            unlimited seconds
Max file size             unlimited            unlimited bytes
Max data size             unlimited            unlimited bytes
Max stack size            8388608              unlimited bytes
Max core file size        unlimited            unlimited bytes
Max resident set          unlimited            unlimited bytes
Max processes             47194                47194 processes
Max open files            1024                 4096 files
Max locked memory         65536                65536 bytes
Max address space         unlimited            unlimited bytes
Max file locks            unlimited            unlimited locks
Max pending signals       47194                47194 signals
Max msgqueue size         819200               819200 bytes
Max nice priority         0                    0
Max realtime priority     0                    0
Max realtime timeout      unlimited            unlimited            us
Core pattern: |/usr/libexec/abrt-hook-ccpp %s %c %p %u %g %t %P %I

Segmentation fault (core dumped)

Closes #537

use s->fields instead of s->stored_fields. extra_null_bitmap is allocated in Table_triggers_list::prepare_record_accessors with respect to virtual fields, so it will not overflow

Closes #1292

This reverts commit 61f370a3
and implements a simpler fix that is straightforward to merge to 10.3.

lock_print_info: Renamed from PrintNotStarted. Dump the
entire contents of trx_sys->mysql_trx_list.

lock_print_info_rw_recovered: Like lock_print_info, but dump
only recovered transactions in trx_sys->rw_trx_list.

lock_print_info_all_transactions(): Dump both trx_sys->mysql_trx_list
and trx_sys->rw_trx_list.

TrxLockIterator, TrxListIterator, lock_rec_fetch_page(): Remove.
This is a partial backport of the 10.3
commit a447980f
which removed the race-condition-prone ability of the InnoDB monitor
to read relevant pages into the buffer pool for some record locks.

In InnoDB error messages, replace the hyperlink URLs to point
to the MariaDB knowledge base.

ut_list_validate(), ut_list_map(): Add variants with const Functor&
so that these functions can be called with an rvalue.

Remove wrapper macros, and add #ifdef UNIV_DEBUG around debug-only code.

MDEV-18429 Consistent non-locking reads do not appear in TRANSACTIONS section of SHOW ENGINE INNODB STATUS

lock_print_info_all_transactions(): print transactions which are started
but not in RW or RO lists.

print_not_started(): Replaces PrintNotStarted. Collect the skipped
transactions.

fil_node_t::read_page0(): Do not replace up-to-date metadata with a
possibly old version of page 0 that is being reread from the file.
A more up-to-date page 0 could still exists in the buffer pool,
waiting to be written back to the file.

BtrBulk::finish(): On error, the B-tree can be corrupted. Only upon
successful completion, it makes sense to validate the index.

The 'SET server_audit_logging ' statements should be logged no matter
what.

Since this limit is imposed by the SHOW_VAR_FUNC_BUFF_SIZE, we just
launch the error message.

Tests for MDEV-15881 Assertion `is_valid_value_slow()' failed in Datetime::Datetime or corrupt data after ALTER with indexed persistent column

The patch for "MDEV-18486 Database crash on a table with indexed virtual column"
fixed MDEV-15881 as well. So adding tests only.

The field roll_node_t::partial holds if and only if
savept has been set. Make savept a pointer.

trx_rollback_start(): Use the semantic type undo_no_t for roll_limit.

PROBLEM
-------
Function innodb_base_col_setup_for_stored() was skipping to store
the base column information for a generated column if the base column
was a "STORED" generated column. This later causes a crash in function
innoabse_col_check_fk() where it says that a generated columns depends
upon two base columns ,but there is information on only one of them.
There was a explicit check barring the stored columns being stored,
which is wrong because the documentation says that a generated stored
column can be a part of a generated column.

FIX
----
Store the information of base column if it is a stored generated column.

#RB21247
Reviewed by: Debarun Banerjee <debarun.banerjee@oracle.com>

Adapt the test case from
mysql/mysql-server@2bbbcddd903626c84c610dd2ed82cc22ee4d6cde
(MySQL 5.7.26).

Problem:
io_getevents() - read asynchronous I/O events from the completion
queue. For each IO event, the res field in io_event tells whether IO
event is succeeded or not. To see if the IO actually succeeded we
always need to check event.res (negative=error,
positive=bytesread/written).
LinuxAIOHandler::collect() doesn't check event.res value for each event.
which leads to incorrect value in n_bytes for IO context (or IO Slot).

Fix:
Added a check for event.res negative value.

RB: 20871
Reviewed by : annamalai.gurusami@oracle.com

MDEV-16518 MYSQL57_GENERATED_FIELD: The code in TABLE_SHARE::init_from_binary_frm_image() is not safe

Problem:
========
The mysqlbinlog tool is leaking memory, causing failures in various tests when
compiling and testing with AddressSanitizer or LeakSanitizer like this:

cmake -DCMAKE_BUILD_TYPE=Debug -DWITH_ASAN:BOOL=ON /path/to/source
make -j$(nproc)
cd mysql-test
ASAN_OPTIONS=abort_on_error=1 ./mtr --parallel=auto

Analysis:
=========
Two types of leaks were observed during above execution.

1) Leak in Log_event::read_log_event(char const*, unsigned int, char const**,
   Format_description_log_event const*, char)
   File: sql/log_event.cc:2150

For all row based replication events the memory which is allocated during
read_log_event is not freed after the event is processed. The event specific
memory has to be retained only when flashback option is enabled with
mysqlbinlog tool. In this case all the events are retained till the end
statement is received and they are processed in reverse order and they are
destroyed. But in the existing code all events are retained irrespective of
flashback mode. Hence the memory leaks are observed.

2) read_remote_annotate_event(unsigned char*, unsigned long, char const**)
   File: client/mysqlbinlog.cc:194

In general the Annotate event is not printed immediately because all
subsequent rbr-events can be filtered away. Instead it will be printed
together with the first not filtered away Table map or the last rbr will be
processed. While reading remote annotate events memory is allocated for event
buffer and event's temp_buf is made to point to the allocated buffer as shown
below.  The TRUE flag is used for doing proper cleanup using free_temp_buf().
i.e at the time of deletion of annotate event its destructor takes care of
clearing the temp_buf.

/*
  Ensure the event->temp_buf is pointing to the allocated buffer.
  (TRUE = free temp_buf on the event deletion)
*/
 event->register_temp_buf((char*)event_buf, TRUE);

But existing code does the following when it receives a remote annotate_event.

if (remote_opt)
  ev->temp_buf= 0;

That is code immediately sets temp_buf=0, because of which free_temp_buf()
call will return empty handed as it has lost the reference to the allocated
temporary buffer. This results in memory leak

Fix:
====
1) If not in flashback mode, destroy the memory for events once they are
processed.

2) Remove the ev->temp_buf=0 code for remote option. Let the proper cleanup to
be done as part of free_temp_buf().

Set table in row ID position mode before using this function.

INSERT and REPLACE served by the same function, so flags (and processing) should be the same.

Normally, the InnoDB master thread executes InnoDB log checkpoints
so frequently that bugs in crash recovery or redo logging can be
hard to reproduce. This is because crash recovery would start replaying
the log only from the latest checkpoint. Because the InnoDB redo log
format only allows saving information for at most 2 latest checkpoints,
and because the log files are written in a circular fashion, it would
be challenging to implement a debug option that would start the redo
log apply from the very start of the redo log file.

It's a micro optimization. On most platforms CPUs has instructions to
compare with 0 fast. DB_SUCCESS is the most popular outcome of functions
and this patch optimized code like (err == DB_SUCCESS)

BtrBulk::finish(): bogus assertion fixed

fil_node_t::read_page0(): corrected usage of os_file_read()

que_eval_sql(): bugus assertion removed. Apparently it checked that
the field was assigned after having been zero-initialized at
object creation.

It turns out that the return type of os_file_read_func() was changed
in mysql/mysql-server@98909cefbc37e54efc6452c7e95bccbf64ac9213 (MySQL 5.7)
from ibool to dberr_t. The reviewer (if there was any) failed to
point out that because of future merges, it could be a bad idea to
change the return type of a function without changing the function name.

This change was applied to MariaDB 10.2.2 in
commit 2e814d47 but the
MariaDB-specific code was not fully adjusted accordingly,
e.g. in fil_node_open_file(). Essentially, code like
!os_file_read(...) became dead code in MariaDB and later
in Mariabackup 10.2, and we could be dealing with an uninitialized
buffer after a failed page read.

InnoDB could return the same list again and again if the buffer
passed to trx_recover_for_mysql() is smaller than the number of
transactions that InnoDB recovered in XA PREPARE state.

We introduce the transaction state TRX_PREPARED_RECOVERED, which
is like TRX_PREPARED, but will be set during trx_recover_for_mysql()
so that each transaction will only be returned once.

Because init_server_components() is invoking ha_recover() twice,
we must reset the state of the transactions back to TRX_PREPARED
after returning the complete list, so that repeated traversals
will see the complete list again, instead of seeing an empty list.
Without this tweak, the test main.tc_heuristic_recover would hang
in MariaDB 10.1.

            failed in compare_order_elements function

The issue here is the function compare_order_lists() is called for the order by list of the window functions
so that those window function that can be computed together are adjacent.
So in the function compare_order_list we iterate over all the elements in the order list of the two functions and
compare the items in their order by clause.
The function compare_order_elements() is called for each item in the
order by clause. This function assumes that all the items that are in the order by list would be of the type
Item::FIELD_ITEM.

The case we have is that we have constants in the order by clause. We should ignore the constant and only compare
items of the type Item::FIELD_ITEM in compare_order_elements()

Problem:
========
The mysqlbinlog tool is leaking memory, causing failures in various tests when
compiling and testing with AddressSanitizer or LeakSanitizer like this:

cmake -DCMAKE_BUILD_TYPE=Debug -DWITH_ASAN:BOOL=ON /path/to/source
make -j$(nproc)
cd mysql-test
ASAN_OPTIONS=abort_on_error=1 ./mtr --parallel=auto rpl.rpl_row_mysqlbinlog

CURRENT_TEST: rpl.rpl_row_mysqlbinlog

Direct leak of 112 byte(s) in 1 object(s) allocated from:
#0 0x4eff87 in __interceptor_malloc (/dev/shm/5.5/client/mysqlbinlog+0x4eff87)
#1 0x60eaab in my_malloc /mariadb/5.5/mysys/my_malloc.c:41:10
#2 0x5300dd in Log_event::read_log_event(char const*, unsigned int, char const**,
   Format_description_log_event const*, char) /mariadb/5.5/sql/log_event.cc:1568:
#3 0x564a9c in dump_remote_log_entries(st_print_event_info*, char const*)
/mariadb/5.5/client/mysqlbinlog.cc:1978:17

Analysis:
========
'mysqlbinlog' tool is being used to read binary log events from a remote server.
While reading binary log, if a fake rotate event is found following actions are
taken.

If 'to-last-log' option is specified, then fake rotate event is processed.
In the absence of 'to-last-log' skip the fake rotate event.

In this skipped case the fake rotate event object is not getting cleaned up
resulting in memory leak.

Fix:
===
Cleanup the fake rotate event.

This issues is already fixed in MariaDB 10.0.23 and higher versions as part of
commit c3018b0f

           with GROUP BY + ORDER BY

The method JOIN::create_postjoin_aggr_table() should not call
call JOIN::add_sorting_to_table() unless the first non-constant join
table is passed as the first parameter to the method.