During merge of innodb code QC invalidation was removed from innodb part but not added to server part.
We decided to keep it in innodb to keep server/engine interface the same.

This only merges MDEV-12253, adapting it to MDEV-12602 which is already
present in 10.2 but not yet in the 10.1 revision that is being merged.

TODO: Error handling in crash recovery needs to be improved.
If a page cannot be decrypted (or read), we should cleanly abort
the startup. If innodb_force_recovery is specified, we should
ignore the problematic page and apply redo log to other pages.
Currently, the test encryption.innodb-redo-badkey randomly fails
like this (the last messages are from cmake -DWITH_ASAN):

2017-05-05 10:19:40 140037071685504 [Note] InnoDB: Starting crash recovery from checkpoint LSN=1635994
2017-05-05 10:19:40 140037071685504 [ERROR] InnoDB: Missing MLOG_FILE_NAME or MLOG_FILE_DELETE before MLOG_CHECKPOINT for tablespace 1
2017-05-05 10:19:40 140037071685504 [ERROR] InnoDB: Plugin initialization aborted at srv0start.cc[2201] with error Data structure corruption
2017-05-05 10:19:41 140037071685504 [Note] InnoDB: Starting shutdown...
i=================================================================
==5226==ERROR: AddressSanitizer: attempting free on address which was not malloc()-ed: 0x612000018588 in thread T0
    #0 0x736750 in operator delete(void*) (/mariadb/server/build/sql/mysqld+0x736750)
    #1 0x1e4833f in LatchCounter::~LatchCounter() /mariadb/server/storage/innobase/include/sync0types.h:599:4
    #2 0x1e480b8 in LatchMeta<LatchCounter>::~LatchMeta() /mariadb/server/storage/innobase/include/sync0types.h:786:17
    #3 0x1e35509 in sync_latch_meta_destroy() /mariadb/server/storage/innobase/sync/sync0debug.cc:1622:3
    #4 0x1e35314 in sync_check_close() /mariadb/server/storage/innobase/sync/sync0debug.cc:1839:2
    #5 0x1dfdc18 in innodb_shutdown() /mariadb/server/storage/innobase/srv/srv0start.cc:2888:2
    #6 0x197e5e6 in innobase_init(void*) /mariadb/server/storage/innobase/handler/ha_innodb.cc:4475:3

Port of mysql changeset by Georgi Kodinov <Georgi.Kodinov@Oracle.com>:

    Bug #34325 : --add-drop-trigger option for mysqldump

    Implemented the --add-drop-trigger option to prepend each
    CREATE TRIGGER in the dump file with DROP TRIGGER.
    The option is off by default. Added a test case.

The new 10.2 has log-slow-admin-statements ON but log-slow-queries OFF,
thus  there is no reason for a warning about it(the change was "by design")

The usage of windows functions when all tables were optimized away
by min/max optimization were not supported. As result a result,
the queries that used window functions with min/max aggregation
over the whole table returned wrong result sets.
The patch fixed this problem.

(from: http://buildbot.askmonty.org/buildbot/builders/p8-rhel6-bintar/builds/820/steps/test/logs/stdio)

Errors like the following indicate a potential endian storage issue:

rocksdb.rocksdb_range                    w1 [ fail ]
        Test ended at 2017-04-27 18:56:11

CURRENT_TEST: rocksdb.rocksdb_range
--- /home/buildbot/maria-slave/p8-rhel6-bintar/build/storage/rocksdb/mysql-test/rocksdb/r/rocksdb_range.result	2017-04-27 17:41:27.740050347 -0400
+++ /home/buildbot/maria-slave/p8-rhel6-bintar/build/storage/rocksdb/mysql-test/rocksdb/r/rocksdb_range.reject	2017-04-27 18:56:11.230050346 -0400
@@ -25,15 +25,15 @@
 select * from t2 force index (a) where a=0;
 pk	a	b
 0	0	0
-1	0	1
-2	0	2
-3	0	3
-4	0	4
-5	0	5
-6	0	6
-7	0	7
-8	0	8
-9	0	9
+16777216	0	1
+33554432	0	2
+50331648	0	3
+67108864	0	4
+83886080	0	5
+100663296	0	6
+117440512	0	7
+134217728	0	8
+150994944	0	9
 # The rest are for code coverage:
 explain
 select * from t2 force index (a) where a=2;
@@ -41,23 +41,23 @@
 1	SIMPLE	t2	ref	a	a	4	const	#
 select * from t2 force index (a) where a=2;
 pk	a	b
-20	2	20
-21	2	21
-22	2	22
-23	2	23
-24	2	24
-25	2	25
-26	2	26
-27	2	27
-28	2	28
-29	2	29
+335544320	2	20
+352321536	2	21
+369098752	2	22
+385875968	2	23
+402653184	2	24
+419430400	2	25
+436207616	2	26
+452984832	2	27
+469762048	2	28
+486539264	2	29
 explain
 select * from t2 force index (a) where a=3 and pk=33;
 id	select_type	table	type	possible_keys	key	key_len	ref	rows	Extra
 1	SIMPLE	t2	const	a	a	8	const,const	#
 select * from t2 force index (a) where a=3 and pk=33;
 pk	a	b
-33	3	33
+553648128	3	33
 select * from t2 force index (a) where a=99 and pk=99;
 pk	a	b
 select * from t2 force index (a) where a=0 and pk=0;
...
Signed-off-by: Daniel Black <daniel.black@au.ibm.com>

        Flags are wrongly set for MBR_CONTAINS/MBR_WITHIN functions.

MDEV-12363 Assertion `0' failed in Type_handler_string_result::make_sort_key(uchar*, Item*, const SORT_FIELD_ATTR*, Sort_param*).

        maybe_null wasn't properly set for Item_json_func_keys.

json_find_path.

        The ..[0] path can be treated wrong on second json_find_path
        call.

JSON_SEARCH with variables.

        Wrong index for the tmp_paths array in
        Item_func_json_search::val_str.

Merged fix into 10.2.

post-fix for 0b52b28b

* restore EXPLAIN output
* remove redundant code

MDEV-12293 Assertion `table->no_keyread || !table->covering_keys.is_set(tab->index) || table->file->keyread == tab->index' failed

sometimes the optimizer starts a keyread on some index
and later (but before any actual index accesses) changes
it to a keyread on a different index

Recompile it for embedded.
Test it for embedded.
Promote it to Gamma

innochecksum uses global variables. great, let's use them all the
way down, instead of passing them as arguments to innodb internals,
conditionally modifying function prototypes with #ifdefs

The bug happened when the specification of a recursive CTE had
no recursive references at the top level of the specification.
In this case the regular processing of derived table references
of the select containing a non-recursive reference to this
recursive CTE misses handling the specification unit.
At the preparation stage any non-recursive reference to a
recursive CTE must be handled after the preparation of the
specification unit for this CTE. So we have to force this
preparation when regular handling of derived tables does not
do it.

This fixes a regression caused by MDEV-12428.
When we introduced a variant of fil_space_acquire() that could
increment space->n_pending_ops after space->stop_new_ops was set,
the logic of fil_check_pending_operations() was broken.

fil_space_t::n_pending_ios: A new field to track read or write
access from the buffer pool routines immediately before a block
write or after a block read in the file system.

fil_space_acquire_for_io(), fil_space_release_for_io(): Similar
to fil_space_acquire_silent() and fil_space_release(), but
modify fil_space_t::n_pending_ios instead of fil_space_t::n_pending_ops.

fil_space_free_low(): Wait for space->n_pending_ios to reach 0,
to avoid accessing freed data in a concurrent thread. Future
calls to fil_space_acquire_for_io() will not find this tablespace,
because it will already have been detached from fil_system.

Adjust a number of places accordingly, and remove some redundant
tablespace lookups.

FIXME: buf_page_check_corrupt() should take a tablespace from
fil_space_acquire_for_io() as a parameter. This will be done
in the 10.1 version of this patch and merged from there.
That depends on MDEV-12253, which has not been merged from 10.1 yet.

Allow setting up binlog_format other then row for session scope for galera/
flashback.

Ref:- Mdev-7322

On several buildbot machines, the test fails like this:

CURRENT_TEST: wsrep.binlog_format
mysqltest: At line 12: query 'SET binlog_format=STATEMENT' failed:
1231: Variable 'binlog_format' can't be set to the value of 'STATEMENT'

ANALYSIS

This is regression caused due to worklog 6742 which
implemented ha_innobase::records() which always
uses clustered index to get the row count. Previously
optimizer chose secondary index which was smaller in
size of clustered index to scan for rows and resulted in
a quicker scan.

FIX

After discussion it was decided to remove this feature in 5.7.

[#rb14040 Approved by Kevin and Oystein ]

- Allow the server to start if innodb force recovery is set to 6
even though change buffer is not empty
Reviewed-by: Jimmy Yang <jimmy.yang@oracle.com>

buf_chunk_not_freed(), logs_empty_and_mark_files_at_shutdown():
Relax debug assertions when innodb_force_recovery=6
implies innodb_read_only.

Problem:
========
 - Drop table assert if innodb_force_recovery is set to 5 or 6.
For innodb_force_recovery 5 and 6, InnoDB doesn't scan the undo log
and it makes the redo rollback segment as NULL. There is no way for
transaction to write any undo log.

 - If innodb_force_recovery is set to 6 then InnoDB does not do the
redo log roll-forward in connection with recovery. In this case,
log_sys will be initalized only and it will not have latest
checkpoint information. Checkpoint is done during shutdown even
innodb_force_recovery is set to 6. So it leads to incorrect
information update in checkpoint header.

Solution:
========
1) Allow drop table only if innodb_force_recovery < 5.
2) Make innodb as read-only if innodb_force_recovery is set to 6.
3) During shutdown, remove the checkpoint if innodb_force_recovery
is set to 6.
Reviewed-by: Jimmy Yang <jimmy.yang@oracle.com>
RB: 15075

Problem:
========
During checkpoint, we are writing all MLOG_FILE_NAME records in one mtr
and parse buffer can't be processed till MLOG_MULTI_REC_END. Eventually parse
buffer exceeds the RECV_PARSING_BUF_SIZE and eventually it overflows.

Fix:
===
1) Break the large mtr if it exceeds LOG_CHECKPOINT_FREE_PER_THREAD into multiple mtr during checkpoint.
2) Move the parsing buffer if we are encountering only MLOG_FILE_NAME
records. So that it will never exceed the RECV_PARSING_BUF_SIZE.
Reviewed-by: Debarun Bannerjee <debarun.bannerjee@oracle.com>
Reviewed-by: Rahul M Malik <rahul.m.malik@oracle.com>
RB: 14743

Description:
===========
Add my_thread_init() and my_thread_exit() for background threads which
initializes and frees the st_my_thread_var structure.

Reviewed-by: Jimmy Yang<jimmy.yang@oracle.com>
RB: 15003

Problem:

During read head, wrong page size is used to calcuate the tablespace size.

Fix:

Use physical page size to calculate tablespace size

Reveiwed-By: Satya Bodapati
RB: 14993

Split the test case so that a server restart is not needed.
Reduce the test cases and use a simpler mechanism for triggering
and waiting for purge.

fil_table_accessible(): Check if a table can be accessed without
enjoying MDL protection.

PROBLEM

When truncating single tablespace tables, we need to scan the entire
buffer pool to remove the pages of the table from the buffer pool.
During this scan and removal dict_sys->mutex is being held ,causing
stalls in other DDL operations.

FIX

Release the dict_sys->mutex during the scan and reacquire it after the
scan. Make sure that purge thread doesn't purge the records of the table
being truncated and background stats collection thread skips the updation
of stats for the table being truncated.

[#rb 14564 Approved by Jimmy and satya ]

srv_sys_t::n_threads_active[]: Protect writes by both the mutex and
by atomic memory access.

srv_active_wake_master_thread_low(): Reliably wake up the master
thread if there is work to do. The trick is to atomically read
srv_sys->n_threads_active[].

srv_wake_purge_thread_if_not_active(): Atomically read
srv_sys->n_threads_active[] (and trx_sys->rseg_history_len),
so that the purge should always be triggered when there is work to do.

trx_commit_in_memory(): Invoke srv_wake_purge_thread_if_not_active()
whenever a transaction is committed. Purge could have been prevented by
the read view of the currently committing transaction, even if it is
a read-only transaction.

trx_purge_add_update_undo_to_history(): Do not wake up the purge.
This is only called by trx_undo_update_cleanup(), as part of
trx_write_serialisation_history(), which in turn is only called by
trx_commit_low() which will always call trx_commit_in_memory().
Thus, the added call in trx_commit_in_memory() will cover also
this use case where a committing read-write transaction added
some update_undo log to the purge queue.

trx_rseg_mem_restore(): Atomically modify trx_sys->rseg_history_len.