MDEV-7847: "Slave worker thread retried transaction 10 time(s) in vain, giving up", followed by replication hanging

This patch fixes a bug in the error handling in parallel replication, when one
worker thread gets a failure and other worker threads processing later
transactions have to rollback and abort.

The problem was with the lifetime of group_commit_orderer objects (GCOs).
A GCO is freed when we register that its last event group has committed. This
relies on register_wait_for_prior_commit() and wait_for_prior_commit() to
ensure that the fact that T2 has committed implies that any earlier T1 has
also committed, and can thus no longer execute mark_start_commit().

However, in the error case, the code was skipping the
register_wait_for_prior_commit() and wait_for_prior_commit() calls. Thus
commit ordering was not guaranteed, and a GCO could be freed too early. Then a
later mark_start_commit() would reference deallocated GCO, which could lead to
lost wakeup (causing slave threads to hang) or other corruption.

This patch makes also the error case respect commit order. This way, also the
error case gets the GCO lifetime correct, and the hang no longer occurs.

When a transaction in parallel replication needs to retry (eg. because of
deadlock kill), first wait for all prior transactions to commit before doing
the retry. This way, we avoid the retry once again conflicting with a prior
transaction, requiring yet another retry.

Without this patch, we saw "in the wild" that transactions had to be retried
more than 10 times to succeed, which exceeds the default
--slave_transaction_retries value and is in any case undesirable.

(We already do this in 10.1 in "optimistic" parallel replication mode; this
patch just makes the code use the same logic for "conservative" mode (only
mode in 10.0)).

Merged revision 5224 from mysql-5.6 and added a test case.
..
revno: 5224
committer: Sergey Glukhov <sergey.glukhov@oracle.com>
branch nick: mysql-5.6
timestamp: Wed 2013-06-19 14:24:08 +0400
message:
    Bug#16620047 INCORRECT QUERY RESULT (AFTER SERVER UPGRADE)

After fix of MDEV-6728, the KILL signal is reset at the start of query
execution. This means that in this test case, we need to wait for the
to-be-killed query to have started; otherwise the kill signal can be lost,
causing the test case to fail.

The test case deliberately crashes the server. If this crash happens in the
middle of a page write, InnoDB crash recovery recovers the page from the
doublewrite buffer, writing a message to the error log that is flagged as a
test failure by mysql-test-run. So add a suppression for this.

  on the multiple table result to obtain the same result on Windows
  and Linux (because files can be retrieved in a different order)
modified:
  storage/connect/mysql-test/connect/r/json.result
  storage/connect/mysql-test/connect/t/json.test

added:
  storage/connect/json.cpp
  storage/connect/json.h
  storage/connect/mysql-test/connect/r/json.result
  storage/connect/mysql-test/connect/std_data/biblio.jsn
  storage/connect/mysql-test/connect/std_data/expense.jsn
  storage/connect/mysql-test/connect/std_data/mulexp3.jsn
  storage/connect/mysql-test/connect/std_data/mulexp4.jsn
  storage/connect/mysql-test/connect/std_data/mulexp5.jsn
  storage/connect/mysql-test/connect/t/json.test
  storage/connect/tabjson.cpp
  storage/connect/tabjson.h
modified:
  storage/connect/CMakeLists.txt
  storage/connect/engmsg.h
  storage/connect/filamtxt.h
  storage/connect/ha_connect.cc
  storage/connect/msgid.h
  storage/connect/mycat.cc
  storage/connect/plgdbsem.h
  storage/connect/tabdos.cpp
  storage/connect/value.cpp
  storage/connect/value.h

added:
  storage/connect/json.cpp
  storage/connect/json.h
  storage/connect/mysql-test/connect/r/json.result
  storage/connect/mysql-test/connect/std_data/biblio.jsn
  storage/connect/mysql-test/connect/std_data/expense.jsn
  storage/connect/mysql-test/connect/std_data/mulexp3.jsn
  storage/connect/mysql-test/connect/std_data/mulexp4.jsn
  storage/connect/mysql-test/connect/std_data/mulexp5.jsn
  storage/connect/mysql-test/connect/t/json.test
  storage/connect/tabjson.cpp
  storage/connect/tabjson.h
modified:
  storage/connect/CMakeLists.txt
  storage/connect/engmsg.h
  storage/connect/filamtxt.h
  storage/connect/ha_connect.cc
  storage/connect/msgid.h
  storage/connect/mycat.cc
  storage/connect/plgdbsem.h
  storage/connect/tabdos.cpp
  storage/connect/value.cpp
  storage/connect/value.h

MDEV-6220 mysqldump will not backup database with --flush-logs parameter and log_error my.cnf parameter defined

some checks were "if [ -n "$err_log" ]", others were "if [ $want_syslog -eq 0 ]",
so when both are set, error log file was only partially initialized.
To avoid this ambiguity we reset want_syslog when error log file is used.

correct the check for PLUGIN_VAR_STR

mtr internally does the following:
1. $default_vardir=....
2. $opt_vardir=$default_vardir unless $opt_vardir;
3. $opt_vardir=realpath $opt_vardir unless IS_WINDOWS
4. if ($opt_vardir eq $default_vardir) { .... use /dev/shm ... }

thus we have to realpath $default_datadir too, otherwise
the comparison logic might be broken

Using a boolean flag for 'there is a RESET MASTER in progress' doesn't
work very well for multiple concurrent RESET MASTER statements.
Changed to a counter.

MDEV-7299 Assertion `m_status == DA_ERROR || m_status == DA_OK' fails on concurrent execution of DDL, queries from I_S, and KILL QUERY

Fix MDL to report an error when a wait was killed, but preserve
the old documented behavior of GET_LOCK() where killing it is not an error.

Also remove race conditions in main.create_or_replace test

revert the code in filesort that conditionally updated 'found_rows',
rely on filesort_limit_arg instead.

MDEV-7186 get_lock() crashes on Windows, main.sp_notembedded and main.trigger_notembedded fail in buildbot

user locks are not affected by lowed_case_table_names

change the test not to use absolute values of Key_blocks_unused.

reset KILL_QUERY when a new query execution is just about to be started

copy the workaround from 5.5

This patch ports the work that facebook has performed
to make innochecksum handle compressed tables.
the basic idea is to use actual innodb-code to perform
checksum verification rather than duplicating in innochecksum.cc.
to make this work, innodb code has been annotated with
lots of #ifndef UNIV_INNOCHECKSUM so that it can be
compiled outside of storage/innobase.

A new testcase is also added that verifies that innochecksum
works on compressed/non-compressed tables.

Merged from commit fabc79d2ea976c4ff5b79bfe913e6bc03ef69d42 
from https://code.google.com/p/google-mysql/

The actual steps to produce this patch are:

    take innochecksum from 5.6.14
    apply changes in innodb from facebook patches needed to make innochecksum compile
    apply changes in innochecksum from facebook patches
    add handcrafted testcase

The referenced facebook patches used are:

    https://github.com/facebook/mysql-5.6/commit/91e25120e75272db4cdbc07d0e45877d9dea5715
    https://github.com/facebook/mysql-5.6/commit/847fe76ea5239b09fa361b023c56e6be76d32046
    https://github.com/facebook/mysql-5.6/commit/1135628a5a9b3412621b93233478f3804bcef51a
    https://github.com/facebook/mysql-5.6/commit/4dbf7c240ce2f08b7d6572d9452c9779ce90641c

mysql-test/suite/maria/insert_select.result:
  Added test case
mysql-test/suite/maria/insert_select.test:
  Added test case
mysys/thr_lock.c:
  Ensure we don't allow concurrent_insert when a read_no_write lock is in use