The problem was that FLUSH TABLE <table_list> would block, 
waiting for all tables with old versions to be removed from 
the table definition cache, rather than waiting for only 
the tables in <table_list>. This could happen if FLUSH TABLE
was used in combination with LOCK TABLES.

With the new MDL code, this problem is no longer repeatable.
Regression test case added to lock.test. This commit contains
no code changes.

revno: 2762 [merge]
committer: Matthias Leich <mleich@mysql.com>
branch nick: mysql-6.0-bugteam-push
timestamp: Wed 2008-08-13 22:05:34 +0200
message:
  Upmerge 5.1 -> 6.0
    ------------------------------------------------------------
    revno: 2497.374.2
    committer: Matthias Leich <mleich@mysql.com>
    branch nick: mysql-5.1-bugteam-push
    timestamp: Wed 2008-08-13 21:44:54 +0200
    message:
      Fix for Bug#37853
          Test "funcs_1.processlist_val_ps" fails in various ways
      + corrections of logic in poll routines
      + minor improvements

This was a deadlock between LOCK TABLES/CREATE DATABASE in one connection
and DROP DATABASE in another. It only happened if the table locked by 
LOCK TABLES was in the database to be dropped. The deadlock is similar
to the one in Bug#48940, but with LOCK TABLES instead of an active
transaction.

The order of events needed to trigger the deadlock was:
1) Connection 1 locks table db1.t1 using LOCK TABLES. It will now
have a metadata lock on the table name.
2) Connection 2 issues DROP DATABASE db1. This will wait inside
the MDL subsystem for the lock on db1.t1 to go away. While waiting, it
will hold the LOCK_mysql_create_db mutex.
3) Connection 1 issues CREATE DATABASE (database name irrelevant).
This will hang trying to lock the same mutex. Since this is the connection
holding the metadata lock blocking Connection 2, we have a deadlock.

This deadlock would also happen for earlier trees without MDL, but 
there DROP DATABASE would wait for a table to be removed from the
table definition cache.

This patch fixes the problem by prohibiting CREATE DATABASE in LOCK TABLES
mode. In the example above, this prevents Connection 1 from hanging trying
to get the LOCK_mysql_create_db mutex. Note that other commands that use
LOCK_mysql_create_db (ALTER/DROP DATABASE) are already prohibited in 
LOCK TABLES mode.

Incompatible change: CREATE DATABASE is now disallowed in LOCK TABLES mode.

Test case added to schema.test.

Bug#45523 "Objects of class base_ilist should not be copyable".
               
Suppress the compiler-generated public copy constructor
and assignment operator of class base_ilist; instead, implement
move_elements_to() function which transfers ownership of elements
from one list to another.

                   INFILE".

Attempts to execute an INSERT statement for a MEMORY table which invoked
a trigger or called a stored function which tried to perform LOW_PRIORITY
update on the table being inserted into, resulted in debug servers aborting
due to an assertion failure. On non-debug servers such INSERTs failed with
"Can't update table t1 in stored function/trigger because it is already used
by statement which invoked this stored function/trigger" as expected.

The problem was that in the above scenario TL_WRITE_CONCURRENT_INSERT
is converted to TL_WRITE inside the thr_lock() function since the MEMORY
engine does not support concurrent inserts. This triggered an assertion
which assumed that for the same table, one thread always requests locks with
higher thr_lock_type value first. When TL_WRITE_CONCURRENT_INSERT is
upgraded to TL_WRITE after the locks have been sorted, this is no longer true.
In this case, TL_WRITE was requested after acquiring a TL_WRITE_LOW_PRIORITY
lock on the table, triggering the assert.

This fix solves the problem by adjusting this assert to take this
scenario into account.

An alternative approach to change handler::store_locks() methods for all engines
which do not support concurrent inserts in such way that
TL_WRITE_CONCURRENT_INSERT is upgraded to TL_WRITE there instead, 
was considered too intrusive.

Commit on behalf of Dmitry Lenev.

This change is supposed to reduce number of ER_LOCK_DEADLOCK
errors which occur when multi-statement transaction encounters
conflicting metadata lock in cases when waiting is possible.

The idea is not to fail ER_LOCK_DEADLOCK error immediately when
we encounter conflicting metadata lock. Instead we release all
metadata locks acquired by current statement and start to wait
until conflicting lock go away. To avoid deadlocks we use simple
empiric which aborts waiting with ER_LOCK_DEADLOCK error if it
turns out that somebody is waiting for metadata locks owned by
this transaction.

This patch also fixes bug #46273 "MySQL 5.4.4 new MDL: Bug#989
is not fully fixed in case of ALTER".

The bug was that concurrent execution of UPDATE or MULTI-UPDATE
statement as a part of multi-statement transaction that already
has used table being updated and ALTER TABLE statement might have
resulted of loss of isolation between this transaction and ALTER
TABLE statement, which manifested itself as changes performed by
ALTER TABLE becoming visible in transaction and wrong binary log
order as a consequence.

This problem occurred when UPDATE or MULTI-UPDATE's wait in
mysql_lock_tables() call was aborted due to metadata lock
upgrade performed by concurrent ALTER TABLE. After such abort all
metadata locks held by transaction were released but transaction
silently continued to be executed as if nothing has happened.

We solve this problem by changing our code not to release all
locks in such case. Instead we release only locks which were
acquired by current statement and then try to reacquire them
by restarting open/lock tables process. We piggyback on simple
deadlock detector implementation since this change has to be
done anyway for it.

3655 Jon Olav Hauglid   2009-10-19
Bug #30977 Concurrent statement using stored function and DROP FUNCTION 
           breaks SBR
Bug #48246 assert in close_thread_table

Implement a fix for:
Bug #41804 purge stored procedure cache causes mysterious hang for many
           minutes
Bug #49972 Crash in prepared statements

The problem was that concurrent execution of DML statements that
use stored functions and DDL statements that drop/modify the same
function might result in incorrect binary log in statement (and
mixed) mode and therefore break replication.

This patch fixes the problem by introducing metadata locking for
stored procedures and functions. This is similar to what is done
in Bug#25144 for views. Procedures and functions now are
locked using metadata locks until the transaction is either
committed or rolled back. This prevents other statements from
modifying the procedure/function while it is being executed. This
provides commit ordering - guaranteeing serializability across
multiple transactions and thus fixes the reported binlog problem.

Note that we do not take locks for top-level CALLs. This means
that procedures called directly are not protected from changes by
simultaneous DDL operations so they are executed at the state they
had at the time of the CALL. By not taking locks for top-level
CALLs, we still allow transactions to be started inside
procedures.

This patch also changes stored procedure cache invalidation.
Upon a change of cache version, we no longer invalidate the entire
cache, but only those routines which we use, only when a statement
is executed that uses them.

This patch also changes the logic of prepared statement validation.
A stored procedure used by a prepared statement is now validated
only once a metadata lock has been acquired. A version mismatch
causes a flush of the obsolete routine from the cache and
statement reprepare.
Incompatible changes:
1) ER_LOCK_DEADLOCK is reported for a transaction trying to access
   a procedure/function that is locked by a DDL operation in
   another connection.

2) Procedure/function DDL operations are now prohibited in LOCK
   TABLES mode as exclusive locks must be taken all at once and
   LOCK TABLES provides no way to specifiy procedures/functions to
   be locked.

Test cases have been added to sp-lock.test and rpl_sp.test.

Work on this bug has very much been a team effort and this patch
includes and is based on contributions from Davi Arnaut, Dmitry
Lenev, Magne Mæhre and Konstantin Osipov.

"HANDLER statements within a transaction might lead to deadlocks".
Introduce a notion of a sentinel to MDL_context. A sentinel
is a ticket that separates all tickets in the context into two
groups: before and after it. Currently we can have (and need) only
one designated sentinel -- it separates all locks taken by LOCK
TABLE or HANDLER statement, which must survive COMMIT and ROLLBACK
and all other locks, which must be released at COMMIT or ROLLBACK.
The tricky part is maintaining the sentinel up to date when
someone release its corresponding ticket. This can happen, e.g.
if someone issues DROP TABLE under LOCK TABLES (generally,
see all calls to release_all_locks_for_name()).
MDL_context::release_ticket() is modified to take care of it.

******
A fix and a test case for Bug#46224 "HANDLER statements within a
transaction might lead to deadlocks".

An attempt to mix HANDLER SQL statements, which are transaction-
agnostic, an open multi-statement transaction,
and DDL against the involved tables (in a concurrent connection) 
could lead to a deadlock. The deadlock would occur when
HANDLER OPEN or HANDLER READ would have to wait on a conflicting
metadata lock. If the connection that issued HANDLER statement
also had other metadata locks (say, acquired in scope of a 
transaction), a classical deadlock situation of mutual wait
could occur.

Incompatible change: entering LOCK TABLES mode automatically
closes all open HANDLERs in the current connection.

Incompatible change: previously an attempt to wait on a lock
in a connection that has an open HANDLER statement could wait
indefinitely/deadlock. After this patch, an error ER_LOCK_DEADLOCK
is produced.

The idea of the fix is to merge thd->handler_mdl_context
with the main mdl_context of the connection, used for transactional
locks. This makes deadlock detection possible, since all waits
with locks are "visible" and available to analysis in a single
MDL context of the connection.

Since HANDLER locks and transactional locks have a different life
cycle -- HANDLERs are explicitly open and closed, and so
are HANDLER locks, explicitly acquired and released, whereas
transactional locks "accumulate" till the end of a transaction
and are released only with COMMIT, ROLLBACK and ROLLBACK TO SAVEPOINT,
a concept of "sentinel" was introduced to MDL_context.
All locks, HANDLER and others, reside in the same linked list.
However, a selected element of the list separates locks with
different life cycle. HANDLER locks always reside at the
end of the list, after the sentinel. Transactional locks are
prepended to the beginning of the list, before the sentinel.
Thus, ROLLBACK, COMMIT or ROLLBACK TO SAVEPOINT, only
release those locks that reside before the sentinel. HANDLER locks
must be released explicitly as part of HANDLER CLOSE statement,
or an implicit close. 
The same approach with sentinel
is also employed for LOCK TABLES locks. Since HANDLER and LOCK TABLES
statement has never worked together, the implementation is
made simple and only maintains one sentinel, which is used either
for HANDLER locks, or for LOCK TABLES locks.

------------------------------------------------------------
revno: 2617.14.26
committer: Vladislav Vaintroub <vvaintroub@mysql.com>
branch nick: mysql-6.0-wtf
timestamp: Wed 2008-11-05 11:19:19 +0100
message:
   CMakeLists.txt files cleanup.
  
  - remove SAFEMALLOC and SAFE_MUTEX definitions that were
  present in *each* CMakeLists.txt. Instead, put them into top level
  MakeLists.txt, but disable on Windows, because
  
  a) SAFEMALLOC does not add any functionality that is not already
  present in Debug C runtime ( and 2 safe malloc one on top of the other
  only unnecessarily slows down the server)
  
  b)SAFE_MUTEX does not work on Windows  and have been
  explicitely  disabled on Windows with #undef previously.  Fortunately,
  ntdll does  pretty good  job identifying l problems with  CRITICAL_SECTIONs.
  (DebugBreak()s on using uninited critical section, unlocking unowned
  critical section)
  
  -Remove occationally used -D_DEBUG (added by compiler
  anyway)
  
  -Remove MAP file generation, it became  obsolete .
  There are many ways to get callstack  of a crash now, with stacktrace in 
  error log , minidump etc

If the handler (or delayed insert) thread failed to lock a table due
to being killed, the "dead" flag was used to notify the connection thread
of this failure. However, with the changes introduced by Bug#45949, 
the handler thread will no longer try to lock the table if it was killed.
This meant that the "dead" flag would not be set, and the connection
thread would not notice that the handler thread had failed.

This could happen with concurrent INSERT DELAYED and FLUSH TABLES.
FLUSH TABLES would kill any active INSERT DELAYED that had opened any
table(s) to be flushed. This could cause the INSERT DELAYED connection
thread to be stuck waiting for the handler thread to lock its table,
while the handler thread would be looping, trying to get the connection
thread to notice the error.

The root of the problem was that the handler thread had both the "dead"
flag and "thd->killed" to indicate that it had been killed. Most places
both were set, but some only set "thd->killed". And 
Delayed_insert::get_local_table() only checked "dead" while waiting for
the table to be locked.

This patch removes the "dead" variable and replaces its usage with
"thd->killed", thereby resolving the issue.

The reason for the deadlock was an improper exit from
MDL_context::wait_for_locks() which caused mysys_var->current_mutex to remain
LOCK_mdl even though LOCK_mdl was no longer held by that connection. 

This could for example lead to a deadlock in the following way:
1) INSERT DELAYED tries to open a table but fails, and trying to recover it
calls wait_for_locks().
2) Due to a pending exclusive request, wait_for_locks() fails and exits without
resetting mysys_var->current_mutex for the delayed insert handler thread. So it
continues to point to LOCK_mdl.
3) The handler thread manages to open a table.
4) A different connection takes LOCK_open and tries to take LOCK_mdl.
5) FLUSH TABLES from a third connection notices that the handler thread has a
table open, and tries to kill it. This involves locking mysys_var->current_mutex
while having LOCK_open locked. Since current_mutex mistakenly points to LOCK_mdl,
we have a deadlock.

This patch makes sure MDL_EXIT_COND() is called before exiting wait_for_locks().
This clears mysys->current_mutex which resolves the issue. 

An assert is added to recover_from_failed_open_table_attempt() after
wait_for_locks() is called, to check that current_mutex is indeed reset.
With this assert in place, existing tests in (e.g.) mdl_sync.test will fail
without this patch.

the fix for Bug#37148, since it is null-merged into 6.0.

This deadlock would occur between two connections A and B if statements
where executed in the following way:
1) Connection A executes a DML statement against table s1.t1 with
autocommit off. This causes a shared metadata lock on s1.t1 to be 
acquired. (With autocommit on, the metadata lock will be dropped once
the statment completes and the deadlock will not occour.)
2) Connection B tries to DROP DATABASE s1. This will block against the
metadata lock connection A holds on s1.t1. While blocking, connection B
will hold the LOCK_mysql_create_db mutex.
3) Connection A tries to ALTER DATABASE s1. This will block when trying
to get LOCK_mysql_create_db mutex held by connection B.
4) Deadlock between DROP DATABASE and ALTER DATABASE (which has autocommit
off).

If Connection A used an explicitly started transaction rather than having
autocommit off, this deadlock did not happen as ALTER DATABASE is 
disallowed inside transactions.

This patch fixes the problem by changing ALTER DATABASE to cause an
implicit commit before executing. This will cause the metadata 
lock on s1.t1 to be dropped, allowing DROP DATABASE to proceed. 
This will in turn cause the LOCK_mysql_create_db mutex to be unlocked, 
allowing ALTER DATABASE to proceed.

Note that SQL commands other than ALTER DATABASE that also use 
LOCK_mysql_create_db, already cause an implicit commit. 

Incompatible change: ALTER DATABASE (and its synonym ALTER SCHEMA)
now cause an implicit commit. This must be reflected in the 
documentation.

Test case added to schema.test.

------------------------------------------------------------
 2599.161.3 Ingo Struewing      2009-07-21
 Bug#20667 - Truncate table fails for a write locked table

 TRUNCATE TABLE was not allowed under LOCK TABLES.

 The patch removes this restriction. mysql_truncate()
 does now handle that case.

-----------------------------------------------------------
2630.28.28 Magne Mahre  2008-12-05
Bug #38661 'all threads hang in "opening tables" or "waiting for table"
            and cpu is at 100%'
                      
Concurrent execution of FLUSH TABLES statement and at least two statements
using the same table might have led to live-lock which caused all three
connections to stall and hog 100% of CPU.
        
tdc_wait_for_old_versions() wrongly assumed that there cannot be a share
with an old version and no used TABLE instances and thus was failing to
perform wait in situation when such old share was cached in MDL subsystem
thanks to a still active metadata lock on the table. So it might have
happened that two or more connections simultaneously executing statements
which involve table being flushed managed to prevent each other from
waiting in this function by keeping shared metadata lock on the table 
constantly active (i.e. one of the statements managed to take/hold this
lock while other statements were calling tdc_wait_for_old_versions()).
Thus they were forcing each other to loop infinitely in open_tables() - 
close_thread_tables_for_reopen() - tdc_wait_for_old_versions() cycle
causing CPU hogging.
        
This patch fixes this problem by removing this false assumption from
tdc_wait_for_old_versions().
 
Note that the problem is specific only for server versions >= 6.0.
        
No test case is submitted for this test, as the test infrastructure
hasn't got the necessary primitives to test the behaviour.  The
manifestation is that throughput will decrease to a low level
(possibly 0) after some time, and stay at that level. Several
transactions will not complete. 
        
Manual testing can be done by running the code submitted by Shane 
Bester attached to the bug report.  If the bug persists, the 
transaction thruput will almost immediately drop to near zero 
(shown as the transaction count output from the test program staying 
on a close to constant value, instead of increasing rapidly).

-----------------------------------------------------------
2497.392.1 Michael Widenius	2008-08-19
Fixes for Bug #38016 Maria: trying to access freed memory when
committing a transaction.
Don't write out states if they haven't changed.

----------------------------------------------------
2736.2.10 Michael Widenius	2008-10-22
Fix for bug#39395 Maria: ma_extra.c:286: maria_extra: 
Assertion `share->reopen == 1' failed

                  
An error occuring in the execution of a stored procedure, called
from do_select is masked, since the error condition is not
propagated back to the caller (join->conds->val_int() returns
a result value, and not an error code)
                  
An explicit check was added to see if the thd error code has been
set, and if so, the loop status is set to the error state.

Backport from 6.0-codebase (revid: 2617.68.31)

          (diagnostics_area)
      
Execution of CREATE TABLE ... SELECT statement was not atomic in
the sense that concurrent statements trying to affect its target
table might have sneaked in between the moment when the table was
created and moment when it was filled according to SELECT clause.
This resulted in inconsistent binary log, unexpected target table
contents. In cases when concurrent statement was a DDL statement
CREATE TABLE ... SELECT might have failed with ER_CANT_LOCK error.
      
In more detail:
Due to premature metadata lock downgrade which occured after CREATE
TABLE SELECT statement created table but before it managed to obtain
table-level lock on it other statements were allowed to open, lock
and change target table in the middle of CREATE TABLE SELECT
execution. This also meant that it was possible that CREATE TABLE
SELECT would wait in mysql_lock_tables() when it was called for newly
created table and that this wait could have been aborted by concurrent
DDL. The latter led to execution of unexpected branch of code and
CREATE TABLE SELECT ending with ER_CANT_LOCK error.
      
The premature downgrade occured because open_table(), which was called
for newly created table, decided that it is OK to downgrade metadata
lock from exclusive to shared since table exists, even although it
was not acquired within this call.
      
This fix ensures that open_table() does not downgrade metadata lock
if it is not acquired during its current invocation.
      
Testing:
The bug is exposed in a race condition, and is thus difficult to
expose in a standard mysql-test-run test case.  Instead, a stress
test using the Random Query Generator (https://launchpad.net/randgen)
will trip the problem occasionally.
      
   % perl  runall.pl \
            --basedir=<build dir> \
             --mysqld=--table-lock-wait-timeout=5 \
             --mysqld=--skip-safemalloc \
             --grammar=conf/maria_bulk_insert.yy \
             --reporters=ErrorLog,Backtrace,WinPackage \
             --mysqld=--log-output=file  \
             --queries=100000 \
             --threads=10 \
             --engine=myisam
      
Note: You will need a debug build to expose the bug
      
When the bug is tripped, the server will abort and dump core.


Backport from 6.0-codebase   (revid: 2617.53.4)

Postfix for Bug#48210 FLUSH TABLES WITH READ LOCK deadlocks
                      against concurrent CREATE PROCEDURE

Rewrote the second test to use DROP PROCEDURE instead of 
CREATE USER as CREATE USER does not work with embedded server.

Bug #48210 FLUSH TABLES WITH READ LOCK deadlocks
           against concurrent CREATE PROCEDURE

This deadlock occured between
a) CREATE PROCEDURE (or other commands listed below)
b) FLUSH TABLES WITH READ LOCK

If the execution of them happened in the following order:
- a) opens a table (e.g. mysql.proc)
- b) locks the global read lock (or GRL)
- a) sleeps inside wait_if_global_read_lock()
- b) increases refresh_version and sleeps waiting 
     for old tables to go away

Note that a) must start waiting on the GRL before FLUSH increases
refresh_version. Otherwise a) won't wait on the GRL and instead
close its tables for reopen, allowing FLUSH to complete and thus
avoid the deadlock.

With this patch the deadlock is avoided by making CREATE PROCEDURE
acquire a protection against global read locks before it starts
executing. This means that FLUSH TABLES WITH READ LOCK will have
to wait until CREATE PROCEDURE completes before acquiring the global
read lock, thereby avoiding the deadlock.

This is implemented by introducing a new SQL command flag called
CF_PROTECT_AGAINST_GRL. Commands marked with this flag will
acquire a GRL protection in the beginning of mysql_execute_command().
This patch adds the flag to CREATE, ALTER and DROP for PROCEDURE
and FUNCTION, as well as CREATE USER, DROP USER, RENAME USER and 
REVOKE ALL. All these commands either call open_grant_tables() or
open_system_table_for_updated() which make them susceptible for
this deadlock.

The patch also adds the CF_PROTECT_AGAINST_GRL flag to a number
of commands that previously acquired GRL protection in their
respective SQLCOM case in mysql_execute_command().

Test case that checks for GRL protection for CREATE PROCEDURE
and CREATE USER added to mdl_sync.test.

 2630.16.14 Sergei Golubchik	2008-08-25
 fixed a crash in partition tests
 introduced by HA_EXTRA_PREPARE_FOR_DROP patch