REPAIR of merge table
Bug #56422 CHECK TABLE run when the table is locked reports
           corruption along with timeout

The crash happened if a table maintenance statement (ANALYZE TABLE,
REPAIR TABLE, etc.) was executed on a MERGE table and opening and 
locking a child table failed. This could for example happen if a child
table did not exist or if a lock timeout happened while waiting for
a conflicting metadata lock to disappear.

Since opening and locking the MERGE table and its children failed,
the tables would be closed and the metadata locks released.
However, TABLE_LIST::table for the MERGE table would still be set,
with its value invalid since the tables had been closed.
This caused the table maintenance statement to try to continue
and upgrade the metadata lock on the MERGE table. But since the lock
already had been released, this caused a segfault.

This patch fixes the problem by setting TABLE_LIST::table to NULL 
if open_and_lock_tables() fails. This prevents maintenance
statements from continuing and trying to upgrade the metadata lock.

The patch includes a 5.5 version of the fix for
Bug #46339 crash on REPAIR TABLE merge table USE_FRM.
This bug caused REPAIR TABLE ... USE_FRM to give an assert 
when used on merge tables.

The patch also enables the CHECK TABLE statement for log tables.
Before, CHECK TABLE for log tables gave ER_CANT_LOCK_LOG_TABLE,
yet still counted the statement as successfully executed.
With the changes to table maintenance statement error handling
in this patch, CHECK TABLE would no longer be considered as
successful in this case. This would have caused upgrade scripts
to mistakenly think that the general and slow logs are corrupted
and have to be repaired. Enabling CHECK TABLES for log tables
prevents this from happening.

Finally, the patch changes the error message from "Corrupt" to
"Operation failed" for a number of issues not related to table
corruption. For example "Lock wait timeout exceeded" and 
"Deadlock found trying to get lock".

Test cases added to merge.test and check.test.

tree for embedded server

Test case for bug #56251 "Deadlock with INSERT
DELAYED and MERGE tables" can't be run against
embedded server. Embedded server converts all
DELAYED INSERTs into ordinary INSERTs and this
test can't work properly if such conversion
happens.

Moved this test from merge.test to delayed.test
which is skipped if test suite is run with
--embedded-server option.

The problem was that RENAME TABLE caused an assert if the system variable
lower_case_table_names was 2 (default on Mac OS X) and the old table name
was given in upper case. This caused lowercase_table2.test to fail.

The assert checks that an exclusive metadata lock is held by the connection
trying to do RENAME TABLE - specificially during updates of table triggers.
The assert was triggered since the check is case sensitive and the lock
was held on the normalized (lower case) version of the table name.

This patch fixes the problem by making sure a normalized version of the
table name is used for the metadata lock check, while using a non-normalized
version of the table name for the rename of trigger files. The same is done
for ALTER TABLE ... RENAME.

Regression testing for the bug itself is already covered by
lowercase_table2.test. Additional coverage added to lowercase_fs_off.test.

Before this fix, the server could crash inside a memcpy when reading data
from the EVENTS_WAITS_CURRENT / HISTORY / HISTORY_LONG  tables.

The root cause is that the length used in a memcpy could be corrupted,
when another thread writes data in the wait record being read.
Reading unsafe data is ok, per design choice, and the code does sanitize
the data in general, but did not sanitize the length given to memcpy.

The fix is to also sanitize the schema name / object name / file name
length when extracting the data to produce a row.

tables".

Attempting to issue an INSERT DELAYED statement for a MERGE
table might have caused a deadlock if it happened as part of
a transaction or under LOCK TABLES, and there was a concurrent
DDL or LOCK TABLES ... WRITE statement which tried to lock one
of its underlying tables.

The problem occurred when a delayed insert handler thread tried
to open a MERGE table and discovered that to do this it had also
to open all underlying tables and hence acquire metadata
locks on them. Since metadata locks on the underlying tables were
not pre-acquired by the connection thread executing INSERT DELAYED,
attempts to do so might lead to waiting. In this case the
connection thread had to wait for the delayed insert thread.
If the thread which was preventing the lock on the underlying table
from being acquired had to wait for the connection thread (due to
this or other metadata locks), a deadlock occurred. 
This deadlock was not detected by the MDL deadlock detector since 
waiting for the handler thread by the connection thread is not
represented in the wait-for graph.

This patch solves the problem by ensuring that the delayed
insert handler thread never tries to open underlying tables 
of a MERGE table. Instead open_tables() is aborted right after
the parent table is opened and a ER_DELAYED_NOT_SUPPORTED 
error is emitted (which is passed to the connection thread and
ultimately to the user).

mysql-test/r/merge.result:
  Added test for bug #56251 "Deadlock with INSERT DELAYED and
  MERGE tables".
mysql-test/t/merge.test:
  Added test for bug #56251 "Deadlock with INSERT DELAYED and
  MERGE tables".
sql/sql_base.cc:
  Changed open_n_lock_single_table() to take prelocking strategy
  as an argument instead of always using DML_prelocking_strategy.
sql/sql_base.h:
  Changed open_n_lock_single_table() to take prelocking strategy
  as an argument instead of always using DML_prelocking_strategy.
  Added a version of this function which is compatible with old
  signature.
sql/sql_insert.cc:
  When opening MERGE table in delayed insert thread stop and emit
  ER_DELAYED_NOT_SUPPORTED right after opening main table and
  before opening underlying tables. This ensures that we won't
  try to acquire metadata lock on underlying tables which might
  lead to a deadlock.
  This is achieved by using special prelocking strategy which
  abort open_tables() process as soon as we discover that we
  have opened table with engine which doesn't support delayed
  inserts.

      
The crash during boot was caused by a DBUG_PRINT statement in fill_schema_schemata() (in
sql_show.cc). This DBUG_PRINT statement contained several instances of %s in the format 
string and for one of these we gave a NULL pointer as the argument. This caused the
call to vsnprintf() to crash when running on Solaris.
      
The fix for this problem is to replace the call to vsnprintf() with my_vsnprintf()
which handles that a NULL pointer is passed as argumens for %s.

This patch also extends my_vsnprintf() to support %i in the format string.

dbug/dbug.c:
  Replace the use of vsnprintf() with my_vsnprintf(). On some platforms
  vsnprintf() did not handle that a NULL pointer was given as an argument
  for a %s in the format string.
include/mysql/service_my_snprintf.h:
  Add support for %i in format string to my_vsnprintf().
strings/my_vsnprintf.c:
  Add support for %i in format string to my_vsnprintf().
unittest/mysys/my_vsnprintf-t.c:
  Add unit tests for %i in format string to my_vsnprintf().

Implemented post review comments.

Added --force to the mysql_upgrade command in the test scripts,
so that the test output does not depends on whether other tests involving an
upgrade have been executed or not in the same test suite execution.

The problem was that issuing XA END when the XA transaction was
already ended, caused an assertion. This assertion tests that 
the server does not try to send OK to the client if there has
already been an error reported. The bug was only noticeable on
debug versions of the server.

The reason for the problem was that the trans_xa_end() function
reported success if the transaction was at XA_IDLE state at the
end regardless of any errors occured during processing of
trans_xa_end(). So if the transaction state was XA_IDLE already,
reported errors would be ignored.

This patch fixes the problem by having trans_xa_end() take into
consideration any reported errors. The patch also fixes a similar
bug with XA PREPARE.

Test case added to xa.test.

Before this fix, the server could crash during shutdown,
due to race conditions, that occured when killing the server.

In particular, the performance schema instrumentation handle,
PSI_server, and the performance schema itself would be cleaned up
too soon, causing race conditions with a running kill server thread.

The specifics of the race condition found are that:
the main thread executing "PSI_server= NULL" can cause crashes in
other threads still running, which are executing
"if (PSI_server != NULL) PSI_server->xxx()"
as part of the performance schema instrumentation.

While the bug was reported for the kill server thread,
in theory the same crash could happen with the signal thread,
as found by code analysis.

The correct fix would be to only shutdown the performance schema
and set PSI_server to NULL after every other thread is guaranteed
to be completed, including the kill_server_thread.

However, due to the existing mysqld server design, this is not the case.
See in particular bug number 56666.

The work around used to fix this race condition is to simply not
perform the call to shutdown_performance_schema() when the server exits,
and to keep the PSI_server pointer unchanged.

This will cause memory leaks to be reported by tools like valgrind,
but no memory leak actually happen because the process is about to exit().

As a result, the file mysql-test/valgrind.supp has been updated
to filter out these false positive messages.

This code has been tested with running in a loop the following
tests in parallel, which have been known to fail with race conditions
in the past:
- rpl_change_master
- binlog_max_extension
- events_restart
- rpl_heartbeat_basic
and no crash of test failure has been seen with the changed code.

Before this fix, it was possible to build the server:
- with the performance schema
- with a dummy implementation of my_atomic (MY_ATOMIC_MODE_DUMMY).

In this case, the resulting binary will just crash,
as this configuration is not supported.

This fix enforces that the build will fail with a compilation error in this
configuration, instead of resulting in a broken binary.

table causes assert failure".

Attempting to use FLUSH TABLE table_list WITH READ LOCK
statement for a MERGE table led to an assertion failure if
one of its children was not present in the list of tables
to be flushed. The problem was not visible in non-debug builds.

The assertion failure was caused by the fact that in such
situations FLUSH TABLES table_list WITH READ LOCK implementation
tried to use (e.g. lock) such child tables without acquiring
metadata lock on them. This happened because when opening tables
we assumed metadata locks on all tables were already acquired
earlier during statement execution and a such assumption was
false for MERGE children.

This patch fixes the problem by ensuring at open_tables() time
that we try to acquire metadata locks on all tables to be opened. 
For normal tables such requests are satisfied instantly since
locks are already acquired for them. For MERGE children metadata
locks are acquired in normal fashion.

Note that FLUSH TABLES merge_table WITH READ LOCK will lock for
read both the MERGE table and its children but will flush only 
the MERGE table. To flush children one has to mention them in table
list explicitly. This is expected behavior and it is consistent with
usage patterns for this statement (e.g. in mysqlhotcopy script).

mysql-test/r/flush.result:
  Added test case for bug #55273 "FLUSH TABLE tm WITH READ LOCK
  for Merge table causes assert failure".
mysql-test/t/flush.test:
  Added test case for bug #55273 "FLUSH TABLE tm WITH READ LOCK
  for Merge table causes assert failure".
sql/sql_base.cc:
  Changed lock_table_names() to support newly introduced
  MYSQL_OPEN_SKIP_SCOPED_MDL_LOCK flag.
sql/sql_base.h:
  Introduced MYSQL_OPEN_SKIP_SCOPED_MDL_LOCK flag for
  open_tables() and lock_table_names() which allows to skip
  acquiring of global and schema-scope locks when SNW, SNRW or
  X metadata locks are acquired.
sql/sql_reload.cc:
  Changed "FLUSH TABLES table_list WITH READ LOCK" code not to
  cause assert about missing metadata locks when MERGE table is
  flushed without one of its underlying tables.
  To achieve this we no longer call open_and_lock_tables() with
  MYSQL_OPEN_HAS_MDL_LOCK flag so this function automatically
  acquires metadata locks on MERGE children if such lock has
  not been already acquired at earlier stage. Instead we call
  this function with MYSQL_OPEN_SKIP_SCOPED_MDL_LOCK flag to
  suppress acquiring of global IX lock in order to keep FLUSH
  TABLES  table_list WITH READ LOCK compatible with FLUSH TABLE
  WITH READ LOCK.
  Also changed implementation to use lock_table_names() function
  for pre-acquiring of metadata locks instead of custom code.
  To implement this change moved setting of open_type member for
  table list elements to parser.
sql/sql_yacc.yy:
  Now we set acceptable type of table for FLUSH TABLES table_list
  WITH READ LOCK at parsing time instead of execution time.

mysys/my_sync.c: In function 'my_sync_dir':
mysys/my_sync.c:103:29: error: unused parameter 'dir_name'
mysys/my_sync.c:103:43: error: unused parameter 'my_flags'
mysys/my_sync.c: In function 'my_sync_dir_by_file':
mysys/my_sync.c:144:37: error: unused parameter 'file_name'
mysys/my_sync.c:144:52: error: unused parameter 'my_flags'

mysys/my_gethwaddr.c: In function 'my_gethwaddr':
mysys/my_gethwaddr.c:67:11: error: pointer targets in assignment differ in signedness

Add a virtual destructor. Class has virtual functions.

With recent changes in the performance schema default sizing parameters,
the memory used by a mysqld binary increased accordingly.

This negatively affects the MTR test suite,
because running several tests in parallel now consumes more ressources.

The fix is to leave the default production values unchanged,
and to configure the MTR environment to limit memory
used when running tests in the test suite, which is ok
because only a few objects are typically used within a test script.

This fix:
- changed the default configuration in MTR to use less memory
- adjusted the performance schema tests accordingly

Note that 1,000 mutex instances was too short and caused test failures
in the past in team trees, so the default used is now 10,000 in MTR.

The amount of memory used by the performance schema itself
can be observed with the statement SHOW ENGINE PERFORMANCE_SCHEMA STATUS

ALTER TABLE on a MERGE table could cause a deadlock with two
other connections if we reached a situation where:

1) A connection doing ALTER TABLE can't upgrade to MDL_EXCLUSIVE on the
parent table, but holds TL_READ_NO_INSERT on the child tables.
2) A connection doing DELETE on a child table can't get TL_WRITE on it
since ALTER TABLE holds TL_READ_NO_INSERT.
3) A connection doing SELECT on the parent table can't get TL_READ on 
the child tables since TL_WRITE is ahead in the lock queue, but holds
MDL_SHARED_READ on the parent table preventing ALTER TABLE from upgrading.

For regular tables, this deadlock is avoided by having ALTER TABLE
take a MDL_SHARED_NO_WRITE metadata lock on the table. This prevents
DELETE from acquiring MDL_SHARED_WRITE on the table before ALTER TABLE
tries to upgrade to MDL_EXCLUSIVE. In the example above, SELECT would
therefore not be blocked by the pending DELETE as DELETE would not be
able to enter TL_WRITE in the table lock queue.

This patch fixes the problem for merge tables by using the same metadata
lock type for child tables as for the parent table. The child tables will
in this case therefore be locked with MDL_SHARED_NO_WRITE, preventing
DELETE from acquiring a metadata lock and enter into the table lock queue.

Change in behavior: By taking the same metadata lock for child tables
as for the parent table, LOCK TABLE on the parent table will now also
implicitly lock the child tables. Since LOCK TABLE on the parent table
now takes more than one metadata lock, it is possible for LOCK TABLE
... WRITE on the parent table or child tables to give ER_LOCK_DEADLOCK
error.

Test case added to mdl_sync.test.
Merge.test/.result has been updated to reflect the change to LOCK TABLE.

The Item_func_str_to_date class wasn't providing correct integer DATETIME
representation as expected. This led to wrong comparison result and didn't
allowed the STR_TO_DATE function to be used with indexes.
Also, STR_TO_DATE function was inconsisted on throwing warnings/errors.
Fixed now.

val_int and result_as_longlong methods were added to the Item_func_str_to_date
class. 

mysql-test/r/func_time.result:
  Test case result adjusted after fixing bug#56271.
mysql-test/r/parser.result:
  Test case result adjusted after fixing bug#56271.
mysql-test/r/select.result:
  A test case result adjusted after fixing bug#56271.
mysql-test/r/strict.result:
  Test case result adjusted after fixing bug#56271.
mysql-test/r/type_datetime.result:
  Added a test case for the bug#56271.
mysql-test/t/strict.test:
  Test case adjusted after fixing bug#56271.
mysql-test/t/type_datetime.test:
  Added a test case for the bug#56271.
sql/item_timefunc.cc:
  Bug#56271: Wrong comparison result with STR_TO_DATE function
  val_int and result_as_longlong methods were added to the Item_func_str_to_date
  class. 
  Item_func_str_to_date::get_date now throws the ER_WRONG_VALUE_FOR_TYPE warning
  on incorrect value.
sql/item_timefunc.h:
  Bug#56271: Wrong comparison result with STR_TO_DATE function
  val_int and result_as_longlong methods were added to the Item_func_str_to_date
  class.

On Solaris with version 3.4.6, the ha_example.so shared library is built
with DTrace and the server is built without DTrace support. This occurs
because dtrace.cmake disables DTrace support for 3.4.6, but still set
HAVE_DTRACE, which causes probes_mysql.h to include probes_mysql_dtrace.h
instead of probes_mysql_nodtrace.h.

This patch fixes this by not setting HAVE_DTRACE on Solaris for GCC 3.4.6.

 create data dir correctly in initial_database target on Windows
 handle case where INSTALL_MYSQLTESTDIR is empty (e.g someone does not want
 to install tests)

          thread-specific variables weren't set when we load error message files.

per-file comments:
  libmysqld/lib_sql.cc
Bug#53251      mysql_library_init fails on second execution with embedded library
      we need to call my_thread_init() once more. Normally it's called at the my_init()
      stage but that doesn't happen on the second my_init() call.

  sql/derror.cc
Bug#53251      mysql_library_init fails on second execution with embedded library
     use default errors for the embedded server.

  sql/mysqld.cc
Bug#53251      mysql_library_init fails on second execution with embedded library
        unregister server errors in clean_up(). Without it the error list contains
        that on the second mysql_server_init() which is not good.

  sql/set_var.cc
Bug#53251      mysql_library_init fails on second execution with embedded library
        sys_var::cleanup() call instead of the destructor

  sql/set_var.h
Bug#53251      mysql_library_init fails on second execution with embedded library
        sys_var::cleanup() introduced instead of the destructor
        
  sql/sys_vars.h
Bug#53251      mysql_library_init fails on second execution with embedded library
        Sys_var_charptr::cleanup() implemented

Merge cleanup, fixed a build warning:

my_getopt.c:156: warning: 'opt_found' may be used uninitialized in this function

        When in embedded-serve mode, mysqltest tried to run '--send' commands in the separate thread.
        That upsets some engines (InnoDB particularly) as the transaction has to be executed in the same
        thread completely. So i implemented some different approach. So we create one separate thread for
        each connection and execute all the queries of this connection inside it. Looks even simpler than it was
        for me.

per-file comments:
  client/mysqltest.cc
Bug#54861      Additional connections not handled properly in mtr --embedded
        Now the connection has one running connection_thread() attached. And sends all the
        query and read-result requests to it.

       made libmysqld/Makefile.am to have same licence as libmysqld/CMakeLists.txt

per-file comments:
  libmysqld/Makefile.am
Bug#54906      Inconsistent license of libmysqld
        Added GPL license header instead of Library GPL.

Problem:
- ORDER BY for utf8mb4_bin, utf16_bin and utf32_bin returned
  results in a wrong order, because old functions
  (supporting only BMP range) were used to handle these collations.
- Additionally, utf16_bin did not sort supplementary characters
  between U+D700 and U+E000, as WL#1213 specification specified.

include/m_ctype.h:
  Adding prototypes.
mysql-test/include/ctype_filesort2.inc:
  Adding a new shared test file.
mysql-test/t/ctype_utf8mb4.test:
  Adding tests.
strings/ctype-ucs2.c:
  - Fixing my_strncoll[sp]_utf16_bin to compare
    binary representation instead of code points,
    to make columns with indexes sort correct.
  - Fixing my_collation_handler_utf32_bin and
    my_collation_handler_utf16_bin to use new
    functions.
strings/ctype-utf8.c:
  - Adding my_strnxfrm[len]_unicode_fill_bin()
    to handle utf8mb4_bin, utf16_bin and utf32_bin,
    using 3 bytes per weight.
    This function also performs special reordering in case of utf16_bin.
  - Fixing my_collation_utf8mb4_bin handler to use the
    new function.

to allow temp table operations) -- prerequisite patch #3.

Rename open_temporary_table() to open_table_uncached().
open_temporary_table() will be introduced in following patches
to open temporary tables for a statement.