With MDEV-12288 and MDEV-13536, the InnoDB purge threads will access
pages more often, causing all sorts of debug assertion failures in
the B-tree code.

Work around this problem by amending the corruption tests with
--innodb-purge-rseg-truncate-frequency=1 --skip-innodb-fast-shutdown
so that everything will be purged before the server
is restarted to deal with the corruption.

This should have been part of MDEV-12288.

trx_undo_t::del_marks: Remove.
Purge needs to process all undo log records in order to
reset the DB_TRX_ID. Before MDEV-12288, it sufficed to only delete
the purgeable delete-marked records, and it ignore other undo log.

trx_rseg_t::needs_purge: Renamed from trx_rseg_t::last_del_marks.
Indicates whether a rollback segment needs to be processed by purge.

TRX_UNDO_NEEDS_PURGE: Renamed from TRX_UNDO_DEL_MARKS.
Indicates whether a rollback segment needs to be processed by purge.
This will be 1 until trx_purge_free_segment() has been invoked.

row_purge_record_func(): Set the is_insert flag for TRX_UNDO_INSERT_REC,
so that the DB_ROLL_PTR will match in row_purge_reset_trx_id().

trx_purge_fetch_next_rec(): Add a comment about row_purge_record_func()
going to set the is_insert flag.

trx_purge_read_undo_rec(): Always attempt to read the undo log record.

trx_purge_get_next_rec(): Do not skip any undo log records.
Even when no clustered index record is going to be removed,
we may want to reset some DB_TRX_ID,DB_ROLL_PTR.

trx_undo_rec_get_cmpl_info(), trx_undo_rec_get_extern_storage(): Remove.

trx_purge_add_undo_to_history(): Set the TRX_UNDO_NEEDS_PURGE flag
so that the resetting will work on undo logs that were originally
created before MDEV-12288 (MariaDB 10.3.1).

trx_undo_roll_ptr_is_insert(), trx_purge_free_segment(): Cleanup
(should be no functional change).

Recording more tests for MDEV-13500 sql_mode=ORACLE: can't create a virtual column with function MOD

Some affected tests require --big-test.
They were forgotten in the main patch.

Corrected an assertion in the constructor for the class Sys_var_flagset.

Fixing Item_func_mod::print() to print "arg1 MOD arg2" instea of "arg1 % arg2"

with window functions (mdev-10855).

This patch just modified the function pushdown_cond_for_derived()
to support this feature.
Some test cases demonstrating this optimization were added to
derived_cond_pushdown.test.

If the server is upgraded from a database that was created
before MDEV-12288, and if the undo logs in the database contain
an incomplete transaction that performed an INSERT operation,
the server would crash when rolling back that transaction.

trx_commit_low(): Relax a too strict transaction. This function
will also be called after completing the rollback of a recovered
transaction.

trx_purge_add_undo_to_history(): Merged from the functions
trx_purge_add_update_undo_to_history() and trx_undo_update_cleanup(),
which are removed. Remove the parameter undo_page, and instead call
trx_undo_set_state_at_finish() to obtain it.

trx_write_serialisation_history(): Treat undo and old_insert equally.
That is, after the rollback (or XA COMMIT) of a recovered transaction
before upgrade, move all logs (both insert_undo and update_undo) to
the purge queue.

Conflicts:
	sql/item_cmpfunc.cc
	storage/innobase/buf/buf0flu.cc
	storage/innobase/include/ut0stage.h
	storage/innobase/row/row0upd.cc

        JSON_EXTRACT behaves specifically in the comparison,
        so we have to implement specific method for that in
        Arg_comparator.

Conflicts:
	sql/item_cmpfunc.cc

developed to cover the case of mdev-13389: "Optimization for equi-joins
of derived tables with window functions".

"Optimization for equi-joins of derived tables with GROUP BY"
should be considered rather as a 'proof of concept'.

The task itself is targeted at an optimization that employs re-writing
equi-joins with grouping derived tables / views into lateral
derived tables. Here's an example of such transformation:
  select t1.a,t.max,t.min
  from t1 [left] join
       (select a, max(t2.b) max, min(t2.b) min from t2
       group by t2.a) as t
       on t1.a=t.a;
=>
  select t1.a,tl.max,tl.min
  from t1 [left] join
       lateral (select a, max(t2.b) max, min(t2.b) min from t2
                where  t1.a=t2.a) as t
       on 1=1;
The transformation pushes the equi-join condition t1.a=t.a into the
derived table making it dependent on table t1. It means that for
every row from t1 a new derived table must be filled out. However
the size of any of these derived tables is just a fraction of the
original derived table t. One could say that transformation 'splits'
the rows used for the GROUP BY operation into separate groups
performing aggregation for a group only in the case when there is
a match for the current row of t1.
Apparently the transformation may produce a query with a better
performance only in the case when
 - the GROUP BY list refers only to fields returned by the derived table
 - there is an index I on one of the tables T used in FROM list of
   the specification of the derived table whose prefix covers the
   the fields from the proper beginning of the GROUP BY list or
   fields that are equal to those fields.
Whether the result of the re-writing can be executed faster depends
on many factors:
  - the size of the original derived table
  - the size of the table T
  - whether the index I is clustering for table T
  - whether the index I fully covers the GROUP BY list.

This patch only tries to improve the chosen execution plan using
this transformation. It tries to do it only when the chosen
plan reaches the derived table by a key whose prefix covers
all the fields of the derived table produced by the fields of
the table T from the GROUP BY list.
The code of the patch does not evaluates the cost of the improved
plan. If certain conditions are met the transformation is applied.

        JSON_EXTRACT behaves specifically in the comparison,
        so we have to implement specific method for that in
        Arg_comparator.

buf_page_io_complete(): Do not test bpage for NULL, because
it is declared (and always passed) as nonnull.

buf_flush_batch(): Remove the constant local variable count=0.

fil_ibd_load(): Use magic comment to suppress -Wimplicit-fallthrough.

ut_stage_alter_t::inc(ulint): Disable references to an unused parameter.

lock_queue_validate(), sync_array_find_thread(), rbt_check_ordering():
Define only in debug builds.

When undo log pages pre-exist from an upgrade, the
TRX_UNDO_PAGE_TYPE could be TRX_UNDO_INSERT==1 (for insert_undo)
TRX_UNDO_UPDATE==2 (for update_undo), instead of the new unified
page type 0 that was introduced in MDEV-12288.

The undo log page type does not really matter much, because the
undo log record type identifies the records independently
of the page type. So, the debug assertions can simply allow any
potential value of the TRX_UNDO_PAGE_TYPE (0, 1, or 2).

MDEV-13475 InnoDB: Failing assertion: lsn == log_sys->lsn || srv_force_recovery == SRV_FORCE_NO_LOG_REDO

At shutdown, we would have lsn == srv_start_lsn == 0 if the server
was started up in --innodb-read-only mode with an old-format redo log.

This regression was caused by MDEV-13430, which skips some redo log
processing at startup when the redo log file format differs (and the
redo log has been determined to be logically empty).

Even though the MDEV-13430 change was introduced in MariaDB 10.2.8,
the MariaDB Server 10.2 series is unaffected by this, because
it will refuse to start up from a version-tagged redo log that is
not tagged to be in the MySQL 5.7.9 or MariaDB 10.2.2 format.
Starting with MariaDB 10.3.1, there are multiple version-tagged
redo log formats.

recv_recovery_from_checkpoint_start(): When skipping an empty
different-format redo log, initialize srv_start_lsn and
recv_sys->recovered_lsn from the redo log file.

The bug is result adding ability to have derived tables inside views.
Fixed checks should be a switch between view/derived or select derived and information schema.

Only a relevant subset of the InnoDB changes was merged.
In particular, two follow-up bug fixes for the bugs that
were introduced in 5.7.18 but not MariaDB 10.2.7 were omitted.
Because MariaDB 10.2.7 omitted the risky change

Bug#23481444 OPTIMISER CALL ROW_SEARCH_MVCC() AND READ THE INDEX
APPLIED BY UNCOMMITTED ROWS

we do not need the follow-up fixes that were introduced in
MySQL 5.6.37 and MySQL 5.7.19:

Bug#25175249 ASSERTION: (TEMPL->IS_VIRTUAL && !FIELD) || ...
Bug#25793677 INNODB: FAILING ASSERTION: CLUST_TEMPL_FOR_SEC || LEN

Analysis:
=========
   During alter table rebuild, InnoDB fails to apply concurrent insert log.
If the insert log record is present across the blocks then apply phase
trying to access the next block without fetching it.

Fix:
====
During virtual column parsing, check whether the record is present
across the blocks before accessing the virtual column information.
Reviewed-by: Jimmy Yang <jimmy.yang@oracle.com>
RB: 16243

Analysis:
========
During alter table rebuild, InnoDB fails to apply concurrent delete log.
Parsing and validation of merge record happens while applying the
log operation on a table. Validation goes wrong for the virtual column.
Validation assumes that virtual column information can't be the end
of the merge record end.

Fix:
====
Virtual column information in the merge record can be end of the merge
record. Virtual column information is written at the end for
row_log_table_delete().

Reviewed-by: Satya Bodapati<satya.bodapati@oracle.com>
RB: 16155

The test is for a bug that was introduced in MySQL 5.7.18
but not MariaDB 10.2, because MariaDB did not merge the change
that was considered incomplete and too risky for a GA release:

Bug#23481444 OPTIMISER CALL ROW_SEARCH_MVCC() AND READ THE INDEX
APPLIED BY UNCOMMITTED ROWS

So, we are only merging the test changes from the bug fix in MySQL 5.7.19,
not any code changes:

commit 4f86aca37d551cc756d9187ec901f8c4a68a0543
Author: Thirunarayanan Balathandayuthapani <thirunarayanan.balathandayuth@oracle.com>
Date:   Wed Apr 26 11:10:41 2017 +0530

    Bug #25793677   INNODB: FAILING ASSERTION: CLUST_TEMPL_FOR_SEC || LEN

MariaDB 10.2 never contained the Oracle change

Bug#23481444 OPTIMISER CALL ROW_SEARCH_MVCC() AND READ THE
INDEX APPLIED BY UNCOMMITTED ROWS

because it was considered risky for a GA release and incomplete.
Remove the references that were added when merging MySQL 5.6.36
to MariaDB 10.0.31, 10.1.24, and 10.2.7.

Analysis:
========

(1) During TRUNCATE of file_per_table tablespace, dict_operation_lock is
released before eviction of dirty pages of a tablespace from the buffer
pool. After eviction, we try to re-acquire
dict_operation_lock (higher level latch) but we already hold lower
level latch (index->lock). This causes latch order violation

(2) Deadlock issue is present if child table is being truncated and it
holds index lock. At the same time, cascade dml happens and it took
dict_operation_lock and waiting for index lock.

Fix:
====
1) Release the indexes lock before releasing the dict operation lock.

2) Ignore the cascading dml operation on the parent table, for the
cascading foreign key, if the child table is truncated or if it is
in the process of being truncated.
Reviewed-by: Jimmy Yang <jimmy.yang@oracle.com>
Reviewed-by: Kevin Lewis <kevin.lewis@oracle.com>
RB: 16122

Issue
====
The issue is that the info message that InnoDB prints when a table
is created with a reference which doesn't exist fills up the log as
it's printed for every insert when foreign_key_checks is disabled.

Fix
===
The fix is to display the message only if foreign_key_checks is
enabled.
Reviewed-by: Jimmy Yang <jimmy.yang@oracle.com>