Since commit baa5a43d
changed the allocation of log_t::files::file_header_bufs[],
a memory overrun could occur when running with a non-default
setting of innodb_log_files_in_group.

log_t::files::create(): Correctly allocate n_files*2048 + 512
bytes of memory, to be aligned at a 512-byte boundary.

For some reason, two replication tests started failing intermittently
after commit 442a6e6b
changed the InnoDB version number from 5.7.21 to 10.3.7.
There is no obvious reference to INNODB_VERSION or
PLUGIN_AUTH_VERSION or PLUGIN_LIBRARY_VERSION in the replication tests.

Can't repeat issue so I just added the test case

We'll be fixing soon shift-reduce conflicts introduced in the new
10.3 syntax (see MDEV-15818 for details) by defining precedence for
a number of tokens (e.g. TIMESTAMP, TRANSACTION_SYM, TEXT_STRING)
and adding "%prec" directives.

Before doing this, it's better to have the existing precedences set properly,
for easier readability and maintainability.

Details:
- Changing precedence of NOT to its proper position (between AND and IS).
  It was wrong. It worked fine only because the relevant grammar reside
  in different separate rules (expr and predicate).

- Moving NOT2_SYM and BINARY to the same line with NEG and ~
  It worked fine because operators !, BINARY, ~ do not conflict
  to each other.

- Fixing associativity of NOT_SYM, NOT2_SYM, BINARY, COLLATE_SYM
  from "right" to "left". They are not dyadic (operate on a single expression
  only). So "%left" or "%right" is needed only to set precedence,
  while associativity does not matter.
  Note, it would be better to use "%precedence" for these tokens
  instead of "%left" though, but we use an old version of Bison on windows,
  which may not support %precedence yet.

This patch does not change behavior. The generated sql_yacc.cc and
sql_yacc_ora.cc are exactly the same before and after this change.

srv_purge_coordinator_thread(): Wait for all purge worker threads
to actually exit. An analysis of a core dump of a hung 10.3 server
revealed that one srv_worker_thread did not exit, even though the
purge coordinator had exited. This caused kill_server_thread and
mysqld_main to wait indefinitely. The main InnoDB shutdown was
never called, because unireg_end() was never called.

Imported the following test case from mysql to MariaDB

1) innodb.alter_kill
2) innodb.alter_foreign_crash
3) innodb.alter_rename_files
4) innodb.analyze_table
5) Appended the case in innodb-online-alter-gis

Problem:
We keep pinning pages in dict_stats_analyze_index_below_cur(),
but doesn't release these pages. When we have a relative small
buffer pool size, and big innodb_stats_persistent_sample_pages,
there will be no free pages for use.

Solution:
Use a separate mtr in dict_stats_analyze_index_below_cur(),
and commit mtr before return.
Reviewed-by: Jimmy Yang <jimmy.yang@oracle.com>
RB: 11362

In debug builds of MySQL, there is an configuration variable
that allows an InnoDB log checkpoint to be initiated:

SET GLOBAL innodb_log_checkpoint_now=ON;

Setting this variable while a table-rebuilding ALTER TABLE is executing
may result in an infinite loop.

checkpoint_now_set(): Account for log_sys->append_on_checkpoint->size().
Note that this function contains race conditions, because it is accessing
fields of log_sys without holding log_sys->mutex. We think that this is
acceptable, because this variable only exists for debugging purposes, in
debug builds of MySQL.

RB: 9947
Reviewed-by: Sunny Bains <sunny.bains@oracle.com>

to innodb.innodb-online-alter-gis

Problem:

The function row_build_index_entry_low() takes a dtuple_t object ('row') and
dict_index_t object ('index') as input and returns a new dtuple_t object
('entry') as output.  The dtuple_t object 'row' that is given as input might
have been constructed from a different dict_index_t object (!= index).  So
when accessing the externally stored data of the given 'row' we need to make
use of the correct index object.

Solution:

Store the page size information in the associated row_ext_t object.

rb#6086 approved by Vasil and Jimmy.

Problem:

Suppose there are two tables in the database related through a foreign key
constraint - the parent table and the child table.  Suppose there is an ALTER
TABLE on the child table which gets interrupted in such a way that the child
table is not available any more.

After crash recovery, an ALTER TABLE on the parent table identifies that its
foreign keys cannot be loaded.  This results in an error and a debug assert.

Solution:

Remove the debug assert and change error to a warning.

rb#8658 approved by Marko.

New added test case: alter_kill in innodb suite.

The server crashes on a SELECT because of space id mismatch. The
mismatch happens if the server crashes during an ALTER TABLE.

There are actually two cases of inconsistency, and three fixes needed
for the InnoDB problems.

We have dictionary data (tablespace or table name) in 3 places:

(a) The *.frm file is for the old table definition.
(b) The InnoDB data dictionary is for the new table definition.
(c) The file system did not rename the tablespace files yet.

In this fix, we will not care if the *.frm file is in sync with the
InnoDB data dictionary and file system. We will concentrate on the
mismatch between (b) and (c).

Two scenarios have been mentioned in this bug report. The simpler one
first:

1. The changes to SYS_TABLES were committed, and MLOG_FILE_RENAME2
records were written in a single mini-transaction commit.
The files were not yet renamed in the file system.
2a. The server is killed, without making a log checkpoint.
3a. The server refuses to start up, because replaying MLOG_FILE_RENAME2
fails.

I failed to repeat this myself. I repeated step 3a with a saved
dataset. The problem seems to be that MLOG_FILE_RENAME2 replay is
incorrectly being skipped when there is no page-redo log or
MLOG_FILE_NAME record for the old name of the tablespace.

FIX#1: Recover the id-to-name mapping also from MLOG_FILE_RENAME2
records when scanning the redo log. It is not necessary to write
MLOG_FILE_NAME records in addition to MLOG_FILE_RENAME2 records for
renaming tablespace files.

The scenario in the original Description involves a log checkpoint:
1. The changes to SYS_TABLES were committed, and MLOG_FILE_RENAME2
records were written in a single mini-transaction commit.
2. A log checkpoint and a server kill was injected.
3. Crash recovery will see no records (other than the MLOG_CHECKPOINT).
4. dict_check_tablespaces_and_store_max_id() will emit a message about
a non-found table #sql-ib22*.
5. A mismatch is triggering the assertion failure.

In my test, at step 4 the SYS_TABLES root page (0:8) contains these 3
records right before the page supremum:
* delete-marked (committed) name=#sql-ib21* record, with space=10.
* name=#sql-ib22*, space=9.
* name=t1, space=10.
space=10 is the rebuilt table (#sql-ib21*.ibd in the file system).
space=9 is the old table (t1.ibd in the file system).

The function dict_check_tablespaces_and_store_max_id() will enter
t1.ibd with space_id=10 into the fil_system cache without noticing
that t1.ibd contains space_id=9, because it invokes
fil_open_single_table_tablespace() with validate=false.

In MySQL 5.6, the space_id from all *.ibd files are being read when
the redo log checkpoint LSN disagrees with the FIL_PAGE_FILE_FLUSH_LSN
in the system tablespace. This field is only updated during a clean
shutdown, after performing the final log checkpoint.

FIX#2: dict_check_tablespaces_and_store_max_id() should pass
validate=true to fil_open_single_table_tablespace() when a non-clean
shutdown is detected, forcing the first page of each *.ibd file to be
read. (We do not want to slow down startup after a normal shutdown.)

With FIX#2, the SELECT would fail to find the table. This would
introduce a regression, because before WL#7142, a copy of the table
was accessible after recovery.

FIX#3: Maintain a list of MLOG_FILE_RENAME2 records that have been
written to the redo log, but not performed yet in the file system.
When performing a checkpoint, re-emit these records to the redo
log. In this way, a mismatch between (b) and (c) should be impossible.

fil_name_process(): Refactored from fil_name_parse(). Adds an item to
the id-to-filename mapping.

fil_name_parse(): Parses and applies a MLOG_FILE_NAME,
MLOG_FILE_DELETE or MLOG_FILE_RENAME2 record. This implements FIX#1.

fil_name_write_rename(): A wrapper function for writing
MLOG_FILE_RENAME2 records.

fil_op_replay_rename(): Apply MLOG_FILE_RENAME2 records. Replaces
fil_op_log_parse_or_replay(), whose logic was moved to fil_name_parse().

fil_tablespace_exists_in_mem(): Return fil_space_t* instead of bool.

dict_check_tablespaces_and_store_max_id(): Add the parameter
"validate" to implement FIX#2.

log_sys->append_on_checkpoint: Extra log records to append in case of
a checkpoint. Needed for FIX#3.

log_append_on_checkpoint(): New function, to update
log_sys->append_on_checkpoint.

mtr_write_log(): New function, to append mtr_buf_t to the redo log.

fil_names_clear(): Append the data from log_sys->append_on_checkpoint
if needed.

ha_innobase::commit_inplace_alter_table(): Add any MLOG_FILE_RENAME2
records to log_sys->append_on_checkpoint(), and remove them once the
files have been renamed in the file system.

mtr_buf_copy_t: A helper functor for copying a mini-transaction log.

rb#6282 approved by Jimmy Yang

While the record type MLOG_UNDO_INIT feels redundant, it does save
redo log volume, and subtle changes to redo log volume appear to
make a big difference for the log_sys.mutex contention.

trx_undo_parse_page_init(): Allow type==0, which is what we write
since MDEV-12288. Parse type in a simpler way; it always was written
as a single byte.

trx_undo_page_init(): Write a MLOG_UNDO_INIT record.

In this issue we hit the assert because we are adding addition fields to the field JOIN::all_fields list. This
is done because  HEAP tables can't index BIT fields so  we need to use an additional hidden field for grouping because later it will be
converted to a LONG field. Original field will remain of the BIT type and will be returned. This happens when we convert DISTINCT to
GROUP BY.

The solution is to take into account the number of such hidden fields that would be added to the field
JOIN::all_fields list while calculating the size of the ref_pointer_array.

Check index capabilities before executing HANDLER READ command.

Thanks to Sergey Vojtovich for feedback and many ideas.

purge_state_t: Remove. The states are replaced with
purge_sys_t::enabled() and purge_sys_t::paused() as follows:
PURGE_STATE_INIT, PURGE_STATE_EXIT, PURGE_STATE_DISABLED: !enabled().
PURGE_STATE_RUN, PURGE_STATE_STOP: paused() distinguishes these.

purge_sys_t::m_paused: Renamed from purge_sys_t::n_stop.
Protected by atomic memory access only, not purge_sys_t::latch.

purge_sys_t::m_enabled: An atomically updated Boolean that
replaces purge_sys_t::state.

purge_sys_t::running: Remove, because it duplicates
srv_sys.n_threads_active[SRV_PURGE].

purge_sys_t::running(): Accessor for srv_sys.n_threads_active[SRV_PURGE].

purge_sys_t::stop(): Renamed from trx_purge_stop().

purge_sys_t::resume(): Renamed from trx_purge_run().
Do not acquire latch; solely rely on atomics.

purge_sys_t::is_initialised(), purge_sys_t::m_initialised: Remove.

purge_sys_t::create(), purge_sys_t::close(): Instead of invoking
is_initialised(), check whether event is NULL.

purge_sys_t::event: Move before latch, so that fields that are
protected by latch can reside on the same cache line with latch.

srv_start_wait_for_purge_to_start(): Merge to the only caller srv_start().

Because the InnoDB implementation in MariaDB has diverged from MySQL,
it is not meaningful to report a MySQL version number for InnoDB
any more. Some examples include:

MariaDB 10.1 (which is based on MySQL 5.6) included encryption and
variable-size page compression before MySQL 5.7 introduced them.
MariaDB 10.2 (based on MySQL 5.7) introduced persistent AUTO_INCREMENT
(MDEV-6076) in a GA release before MySQL 8.0.
MariaDB 10.3 (based on MySQL 5.7) introduced instant ADD COLUMN
(MDEV-11369) before MySQL.

All of these features use a different implementation and file format.
Also, some features were never merged from MySQL 5.7, and thus MariaDB
is not affected by related bugs. Examples include CREATE TABLESPACE
and the reimplementation of the partitioning engine.

row_trx_id_offset(): Add the missing rec_offs_init() call.
This was broken by commit 97e51d24.

Before invoking fil_space_t::acquire(), check space->is_stopping()
to ensure that the tablespace actually is accessible. This fixes
a regression introduced by MDEV-15983.

fil_space_next(): Remove some duplicated code for prev_space==NULL,
and check is_stopping() also on the first tablespace.

i_s_tablespaces_encryption_fill_table(),
i_s_tablespaces_scrubbing_fill_table(): Check is_stopping().

These directives did not have any affect.

All these affected lines:

1.  | INTERVAL_SYM expr interval '+' expr

2.  | INTERVAL_SYM '(' expr ',' expr ')'

3.  | INTERVAL_SYM '(' expr ',' expr ',' expr_list ')'

4.  | DATE_ADD_INTERVAL '(' expr ',' INTERVAL_SYM expr interval ')'

5.  | DATE_SUB_INTERVAL '(' expr ',' INTERVAL_SYM expr interval ')'

define the grammar unambiguously.

"bison -v sql_yacc.yy" produces exactly the same sql_yacc.output
before and after the change.

It caused compile failures:

commit 6620fbd6
Author: Teodor Mircea Ionita <teodor@mariadb.org>
Date:   Thu May 10 12:23:35 2018 +0300

    MDEV-15778: On macOS pthread_t is opaque, requires explicit cast

    On macOS pthread id is a pointer to struct _opaque_pthread_t type,
    requires explicit cast to ulint which in turn is size_t;
    Was failing with Clang 9.1.0 Debug build on macOS 10.13.4:

    sync0policy.h:53:4: error: cannot initialize a member subobject of type 'ulint'
    (aka 'unsigned long') with an rvalue of type 'os_thread_id_t' (aka '_opaque_pthread_t *')
    m_thread_id(os_thread_id_t(ULINT_UNDEFINED))
                            ^           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    sync0policy.h:79:4: error: cannot initialize a parameter of type 'int64' (aka 'long long') with
    an rvalue of type 'os_thread_id_t' (aka '_opaque_pthread_t *')
    my_atomic_storelint(&m_thread_id, os_thread_get_curr_id());

Problem was that verify_constraints() didn't check if
there was an error as part of evaluating constraints
(can happen in strict mode).

In one-row-insert the error was ignored when using
binary logging as binary logging clear errors if insert
succeeded. In multi-row-insert the error was noticed
for the second row.

After this fix one will get an error for both one and
multi-row inserts if the constraints generates a warning
in strict mode.

followup for 21bcfeb9

MDEV-15576: Server crashed in Cached_item_str::cmp / sortcmp or Assertion `item->null_value' failed in Type_handler_temporal_result::make_sort_key upon SELECT with NULLIF and ROLLUP

Fixed null_value processing and is_null() usage.

Problem was that I in a previous patch enabled ifdef:ed code intended for
10.3 but that never worked.

Reverted to original code

MDEV-654 Assertion `share->now_transactional' failed in flush_log_for_bitmap on concurrent workload with Aria tables

I was able to repeat the problem with old version of randgen

Reason for crash:

- It's not safe to change share->now_transactional if there are changed
bitmaps in the pagecache as flushing these can cause redo-entries and
the bitmap flush code checks that share->now_transactional is set.

Fixed by flushing bitmaps in _ma_tmp_disable_logging_for_table() before
we set share->now_transactional to 0

Part one, non-temporary tables.

Rrenaming a column can make destructive changes
to the TABLE. This TABLE cannot be used anymore
and needs to be reopened even if ALTER TABLE
was aborted with an error.

MDEV-14750 Valgrind Invalid read, ASAN heap-use-after-free in Item_ident::print upon SHOW CREATE on partitioned table

items in the partitioning function were taking
the table name from the table's field
(in set_field(from_field) in Item_field::fix_fields)
and field's table_name is TABLE::alias.

But alias is changed for every statement, and
can be realloced if next statement uses a longer
alias. But partitioning items are fixed once
and live as long as the TABLE does. So if
an alias is realloced, pointers to the old
alias string will become invalid.

Fix partitioning item table_name to point to
the actual table name instead.

rename to post_fix_fields_part_expr_processor()
because it's only used after fix_fields in
fix_fields_part_func() and can be used for
various post-fix_fields fixups