condition variable per context instead of one mutex and one conditional
variable for the whole subsystem.

This should increase concurrency in this subsystem.

It also opens the way for further changes which are necessary to solve
such bugs as bug #46272 "MySQL 5.4.4, new MDL: unnecessary deadlock"
and bug #37346 "innodb does not detect deadlock between update and alter
table".

Two other notable changes done by this patch:

- MDL subsystem no longer implicitly acquires global intention exclusive
  metadata lock when per-object metadata lock is acquired. Now this has
  to be done by explicit calls outside of MDL subsystem.
- Instead of using separate MDL_context for opening system tables/tables
  for purposes of I_S we now create MDL savepoint in the main context
  before opening tables and rollback to this savepoint after closing
  them. This means that it is now possible to get ER_LOCK_DEADLOCK error
  even not inside a transaction. This might happen in unlikely case when
  one runs DDL on one of system tables while also running DDL on some
  other tables. Cases when this ER_LOCK_DEADLOCK error is not justified
  will be addressed by advanced deadlock detector for MDL subsystem which
  we plan to implement.

mysql-test/include/handler.inc:
  Adjusted handler_myisam.test and handler_innodb.test to the fact that
  exclusive metadata locks on tables are now acquired according to
  alphabetical order of fully qualified table names instead of order
  in which tables are mentioned in statement.
mysql-test/r/handler_innodb.result:
  Adjusted handler_myisam.test and handler_innodb.test to the fact that
  exclusive metadata locks on tables are now acquired according to
  alphabetical order of fully qualified table names instead of order
  in which tables are mentioned in statement.
mysql-test/r/handler_myisam.result:
  Adjusted handler_myisam.test and handler_innodb.test to the fact that
  exclusive metadata locks on tables are now acquired according to
  alphabetical order of fully qualified table names instead of order
  in which tables are mentioned in statement.
mysql-test/r/mdl_sync.result:
  Adjusted mdl_sync.test to the fact that exclusive metadata locks on
  tables are now acquired according to alphabetical order of fully
  qualified table names instead of order in which tables are mentioned
  in statement.
mysql-test/t/mdl_sync.test:
  Adjusted mdl_sync.test to the fact that exclusive metadata locks on
  tables are now acquired according to alphabetical order of fully
  qualified table names instead of order in which tables are mentioned
  in statement.
sql/events.cc:
  Instead of using separate MDL_context for opening system tables we now
  create MDL savepoint in the main context before opening such tables
  and rollback to this savepoint after closing them. To support this
  change methods of THD responsible for saving/restoring open table
  state were changed to use Open_tables_backup class which in addition
  to Open_table_state has a member for this savepoint. As result code
  opening/closing system tables was changed to use Open_tables_backup
  instead of Open_table_state class as well.
sql/ha_ndbcluster.cc:
  Since manipulations with open table state no longer install proxy
  MDL_context it does not make sense to perform them in order to
  satisfy assert in mysql_rm_tables_part2(). Removed them per agreement
  with Cluster team. This has not broken test suite since scenario in
  which deadlock can occur and assertion fails is not covered by tests.
sql/lock.cc:
  MDL subsystem no longer implicitly acquires global intention exclusive
  metadata lock when per-object exclusive metadata lock is acquired.
  Now this has to be done by explicit calls outside of MDL subsystem.
sql/log.cc:
  Instead of using separate MDL_context for opening system tables we now
  create MDL savepoint in the main context before opening such tables
  and rollback to this savepoint after closing them. To support this
  change methods of THD responsible for saving/restoring open table
  state were changed to use Open_tables_backup class which in addition
  to Open_table_state has a member for this savepoint. As result code
  opening/closing system tables was changed to use Open_tables_backup
  instead of Open_table_state class as well.
sql/mdl.cc:
  Changed metadata locking subsystem to use mutex per lock and condition
  variable per context instead of one mutex and one conditional variable
  for the whole subsystem.
  Changed approach to handling of global metadata locks. Instead of
  implicitly acquiring intention exclusive locks when user requests
  per-object upgradeable or exclusive locks now we require them to be
  acquired explicitly in the same way as ordinary metadata locks.
  In fact global lock are now ordinary metadata locks in new GLOBAL
  namespace.
  
  To implement these changes:
  - Removed LOCK_mdl mutex and COND_mdl condition variable.
  - Introduced MDL_lock::m_mutex mutexes which protect individual lock
    objects.
  - Replaced mdl_locks hash with MDL_map class, which has hash for
    MDL_lock objects as a member and separate mutex which protects this
    hash. Methods of this class allow to find(), find_or_create() or
    remove() MDL_lock objects in concurrency-friendly fashion (i.e.
    for most common operation, find_or_create(), we don't acquire
    MDL_lock::m_mutex while holding MDL_map::m_mutex. Thanks to MikaelR
    for this idea and benchmarks!). Added three auxiliary members to
    MDL_lock class (m_is_destroyed, m_ref_usage, m_ref_release) to
    support this concurrency-friendly behavior.
  - Introduced MDL_context::m_ctx_wakeup_cond condition variable to be
    used for waiting until this context's pending request can be
    satisfied or its thread has to perform actions to resolve potential
    deadlock. Context which want to wait add ticket corresponding to the
    request to an appropriate queue of waiters in MDL_lock object so
    they can be noticed when other contexts change state of lock and be
    awaken by them by signalling on MDL_context::m_ctx_wakeup_cond.
    As consequence MDL_ticket objects has to be used for any waiting
    in metadata locking subsystem including one which happens in
    MDL_context::wait_for_locks() method.
    Another consequence is that MDL_context is no longer copyable and
    can't be saved/restored when working with system tables.
  - Made MDL_lock an abstract class, which delegates specifying exact
    compatibility matrix to its descendants. Added MDL_global_lock child
    class for global lock (The old is_lock_type_compatible() method
    became can_grant_lock() method of this class). Added MDL_object_lock
    class to represent per-object lock (The old MDL_lock::can_grant_lock()
    became its method). Choice between two classes happens based on MDL
    namespace in MDL_lock::create() method.
  - Got rid of MDL_lock::type member as its meaning became ambigous for
    global locks.
  - To simplify waking up of contexts waiting for lock split waiting queue
    in MDL_lock class in two queues. One for pending requests for exclusive
    (including intention exclusive) locks and another for requests for
    shared locks.
  - Added virtual wake_up_waiters() method to MDL_lock, MDL_global_lock and
    MDL_object_lock classes which allows to wake up waiting contexts after
    state of lock changes. Replaced old duplicated code with calls to this
    method.
  - Adjusted MDL_context::try_acquire_shared_lock()/exclusive_lock()/
    global_shared_lock(), MDL_ticket::upgrade_shared_lock_to_exclusive_lock()
    and MDL_context::release_ticket() methods to use MDL_map and
    MDL_lock::m_mutex instead of single LOCK_mdl mutex and wake up
    waiters according to the approach described above. The latter method
    also was renamed to MDL_context::release_lock().
  - Changed MDL_context::try_acquire_shared_lock()/exclusive_lock() and
    release_lock() not to handle global locks. They are now supposed to
    be taken explicitly like ordinary metadata locks.
  - Added helper MDL_context::try_acquire_global_intention_exclusive_lock()
    and acquire_global_intention_exclusive_lock() methods.
  - Moved common code from MDL_context::acquire_global_shared_lock() and
    acquire_global_intention_exclusive_lock() to new method -
    MDL_context::acquire_lock_impl().
  - Moved common code from MDL_context::try_acquire_shared_lock(),
    try_acquire_global_intention_exclusive_lock()/exclusive_lock()
    to MDL_context::try_acquire_lock_impl().
  - Since acquiring of several exclusive locks can no longer happen under
    single LOCK_mdl mutex the approach to it had to be changed. Now we do
    it in one by one fashion. This is done in alphabetical order to avoid
    deadlocks. Changed MDL_context::acquire_exclusive_locks() accordingly
    (as part of this change moved code responsible for acquiring single
    exclusive lock to new MDL_context::acquire_exclusive_lock_impl()
    method).
  - Since we no longer have single LOCK_mdl mutex which protects all
    MDL_context::m_is_waiting_in_mdl members using these members to
    determine if we have really awaken context holding conflicting
    shared lock became inconvinient. Got rid of this member and changed
    notify_shared_lock() helper function and process of acquiring
    of/upgrading to exclusive lock not to rely on such information.
    Now in MDL_context::acquire_exclusive_lock_impl() and
    MDL_ticket::upgrade_shared_lock_to_exclusive_lock() we simply
    re-try to wake up threads holding conflicting shared locks after
    small time out.
  - Adjusted MDL_context::can_wait_lead_to_deadlock() and
    MDL_ticket::has_pending_conflicting_lock() to use per-lock
    mutexes instead of LOCK_mdl. To do this introduced
    MDL_lock::has_pending_exclusive_lock() method.
sql/mdl.h:
  Changed metadata locking subsystem to use mutex per lock and condition
  variable per context instead of one mutex and one conditional variable
  for the whole subsystem. In order to implement this change:
  
  - Added MDL_key::cmp() method to be able to sort MDL_key objects
    alphabetically. Changed length fields in MDL_key class to uint16
    as 16-bit is enough for length of any key.
  - Changed MDL_ticket::get_ctx() to return pointer to non-const
    object in order to be able to use MDL_context::awake() method
    for such contexts.
  - Got rid of unlocked versions of can_wait_lead_to_deadlock()/
    has_pending_conflicting_lock() methods in MDL_context and
    MDL_ticket. We no longer has single mutex which protects all
    locks. Thus one always has to use versions of these methods
    which acquire per-lock mutexes.
  - MDL_request_list type of list now counts its elements.
  - Added MDL_context::m_ctx_wakeup_cond condition variable to be used
    for waiting until this context's pending request can be satisfied
    or its thread has to perform actions to resolve potential deadlock.
    Added awake() method to wake up context from such wait.
    Addition of condition variable made MDL_context uncopyable.
    As result we no longer can save/restore MDL_context when working
    with system tables. Instead we create MDL savepoint before opening
    those tables and rollback to it once they are closed.
  - MDL_context::release_ticket() became release_lock() method.
  - Added auxiliary MDL_context::acquire_exclusive_lock_impl() method
    which does all necessary work to acquire exclusive lock on one object
    but should not be used directly as it does not enforce any asserts
    ensuring that no deadlocks are possible.
  - Since we no longer need to know if thread trying to acquire exclusive
    lock managed to wake up any threads having conflicting shared locks
    (as, anyway, we will try to wake up such threads again shortly)
  - MDL_context::m_is_waiting_in_mdl member became unnecessary and
    notify_shared_lock() no longer needs to be friend of MDL_context.
  
  Changed approach to handling of global metadata locks. Instead of
  implicitly acquiring intention exclusive locks when user requests
  per-object upgradeable or exclusive locks now we require them to be
  acquired explicitly in the same way as ordinary metadata locks.
  
  - Added new GLOBAL namespace for such locks.
  - Added new type of lock to be requested MDL_INTENTION_EXCLISIVE.
  - Added MDL_context::try_acquire_global_intention_exclusive_lock()
    and acquire_global_intention_exclusive_lock() methods.
  - Moved common code from MDL_context::acquire_global_shared_lock()
    and acquire_global_intention_exclusive_lock() to new method -
    MDL_context::acquire_lock_impl().
  - Moved common code from MDL_context::try_acquire_shared_lock(),
    try_acquire_global_intention_exclusive_lock()/exclusive_lock()
    to MDL_context::try_acquire_lock_impl().
  - Added helper MDL_context::is_global_lock_owner() method to be
    able easily to find what kind of global lock this context holds.
  - MDL_context::m_has_global_shared_lock became unnecessary as
    global read lock is now represented by ordinary ticket.
  - Removed assert in MDL_context::set_lt_or_ha_sentinel() which became
    false for cases when we execute LOCK TABLES under global read lock
    mode.
sql/mysql_priv.h:
  Instead of using separate MDL_context for opening system tables we now
  create MDL savepoint in the main context before opening such tables
  and rollback to this savepoint after closing them. To support this
  change methods of THD responsible for saving/restoring open table
  state were changed to use Open_tables_backup class which in addition
  to Open_table_state has a member for this savepoint. As result calls
  opening/closing system tables were changed to use Open_tables_backup
  instead of Open_table_state class as well.
sql/sp.cc:
  Instead of using separate MDL_context for opening system tables we now
  create MDL savepoint in the main context before opening such tables
  and rollback to this savepoint after closing them. To support this
  change methods of THD responsible for saving/restoring open table
  state were changed to use Open_tables_backup class which in addition
  to Open_table_state has a member for this savepoint. As result code
  opening/closing system tables was changed to use Open_tables_backup
  instead of Open_table_state class as well.
sql/sp.h:
  Instead of using separate MDL_context for opening system tables we now
  create MDL savepoint in the main context before opening such tables
  and rollback to this savepoint after closing them. To support this
  change methods of THD responsible for saving/restoring open table
  state were changed to use Open_tables_backup class which in addition
  to Open_table_state has a member for this savepoint. As result code
  opening/closing system tables was changed to use Open_tables_backup
  instead of Open_table_state class as well.
sql/sql_base.cc:
  close_thread_tables():
    Since we no longer use separate MDL_context for opening system
    tables we need to avoid releasing all transaction locks when
    closing system table. Releasing metadata lock on system table
    is now responsibility of THD::restore_backup_open_tables_state().
  open_table_get_mdl_lock(),
  Open_table_context::recover_from_failed_open():
    MDL subsystem no longer implicitly acquires global intention exclusive
    metadata lock when per-object upgradable or exclusive metadata lock is
    acquired. So this have to be done explicitly from these calls.
    Changed Open_table_context class to store MDL_request object for
    global intention exclusive lock acquired when opening tables.
  open_table():
    Do not release metadata lock if we have failed to open table as
    this lock might have been acquired by one of previous statements
    in transaction, and therefore should not be released.
  open_system_tables_for_read()/close_system_tables()/
  open_performance_schema_table():
    Instead of using separate MDL_context for opening system tables we now
    create MDL savepoint in the main context before opening such tables
    and rollback to this savepoint after closing them. To support this
    change methods of THD responsible for saving/restoring open table
    state were changed to use Open_tables_backup class which in addition
    to Open_table_state has a member for this savepoint. As result code
    opening/closing system tables was changed to use Open_tables_backup
    instead of Open_table_state class as well.
  close_performance_schema_table():
    Got rid of duplicated code.
sql/sql_class.cc:
  Instead of using separate MDL_context for opening system tables we now
  create MDL savepoint in the main context before opening such tables
  and rollback to this savepoint after closing them. To support this
  change methods of THD responsible for saving/restoring open table
  state were changed to use Open_tables_backup class which in addition
  to Open_table_state has a member for this savepoint. Also releasing
  metadata lock on system table is now responsibility of
  THD::restore_backup_open_tables_state().
  Adjusted assert in THD::cleanup() to take into account fact that now
  we also use MDL sentinel for global read lock.
sql/sql_class.h:
  Instead of using separate MDL_context for opening system tables we now
  create MDL savepoint in the main context before opening such tables
  and rollback to this savepoint after closing them. As result:
  - 'mdl_context' member was moved out of Open_tables_state to THD class.
    enter_locked_tables_mode()/leave_locked_tables_mode() had to follow.
  - Methods of THD responsible for saving/restoring open table state were
    changed to use Open_tables_backup class which in addition to
    Open_table_state has a member for this savepoint.
  Changed Open_table_context class to store MDL_request object for
  global intention exclusive lock acquired when opening tables.
sql/sql_delete.cc:
  MDL subsystem no longer implicitly acquires global intention exclusive
  metadata lock when per-object exclusive metadata lock is acquired.
  Now this has to be done by explicit calls outside of MDL subsystem.
sql/sql_help.cc:
  Instead of using separate MDL_context for opening system tables we now
  create MDL savepoint in the main context before opening such tables
  and rollback to this savepoint after closing them. To support this
  change methods of THD responsible for saving/restoring open table
  state were changed to use Open_tables_backup class which in addition
  to Open_table_state has a member for this savepoint. As result code
  opening/closing system tables was changed to use Open_tables_backup
  instead of Open_table_state class as well.
sql/sql_parse.cc:
  Adjusted assert reload_acl_and_cache() to the fact that global read
  lock now takes full-blown metadata lock.
sql/sql_plist.h:
  Added support for element counting to I_P_List list template.
  One can use policy classes to specify if such counting is needed
  or not needed for particular list.
sql/sql_show.cc:
  Instead of using separate MDL_context for opening tables for I_S
  purposes we now create MDL savepoint in the main context before
  opening tables and rollback to this savepoint after closing them.
  To support this and similar change for system tables methods of
  THD responsible for saving/restoring open table state were changed
  to use Open_tables_backup class which in addition to Open_table_state
  has a member for this savepoint. As result code opening/closing tables
  for I_S purposes was changed to use Open_tables_backup instead of
  Open_table_state class as well.
sql/sql_table.cc:
  mysql_rm_tables_part2():
    Since now global intention exclusive metadata lock is ordinary
    metadata lock we no longer can rely that by releasing MDL locks
    on all tables we will release all locks acquired by this routine.
    So in non-LOCK-TABLES mode we have to release all locks acquired
    explicitly.
  prepare_for_repair(), mysql_alter_table():
    MDL subsystem no longer implicitly acquires global intention
    exclusive metadata lock when per-object exclusive metadata lock
    is acquired. Now this has to be done by explicit calls outside of
    MDL subsystem.
sql/tztime.cc:
  Instead of using separate MDL_context for opening system tables we now
  create MDL savepoint in the main context before opening such tables
  and rollback to this savepoint after closing them. To support this
  change methods of THD responsible for saving/restoring open table
  state were changed to use Open_tables_backup class which in addition
  to Open_table_state has a member for this savepoint. As result code
  opening/closing system tables was changed to use Open_tables_backup
  instead of Open_table_state class as well.
  Also changed code not to use special mechanism for open system tables
  when it is not really necessary.

           Diagnostics_area::set_ok_status at PREPARE

The problem occured during processing of stored routines. 
Routines are loaded from mysql.proc, parsed and put into the sp cache by
sp_cache_routine().  The assert occured because the return value from
sp_cache_routine() was not checked for top level CALLs. This meant that any
errors during sp_cache_routine() went unoticed and triggered the assert when
my_ok() was later called.

This is a regression introduced by the patch for Bug#30977, only visible in
source trees with MDL and using debug builds of the server.

This patch fixes the problem by checking the return value from sp_cache_routine() 
for top level CALLs and propagating any errors similar to what is done for other 
calls to sp_cache_routine().

No test case added.

The problem was that FLUSH TABLE <table_list> would block, 
waiting for all tables with old versions to be removed from 
the table definition cache, rather than waiting for only 
the tables in <table_list>. This could happen if FLUSH TABLE
was used in combination with LOCK TABLES.

With the new MDL code, this problem is no longer repeatable.
Regression test case added to lock.test. This commit contains
no code changes.

revno: 2762 [merge]
committer: Matthias Leich <mleich@mysql.com>
branch nick: mysql-6.0-bugteam-push
timestamp: Wed 2008-08-13 22:05:34 +0200
message:
  Upmerge 5.1 -> 6.0
    ------------------------------------------------------------
    revno: 2497.374.2
    committer: Matthias Leich <mleich@mysql.com>
    branch nick: mysql-5.1-bugteam-push
    timestamp: Wed 2008-08-13 21:44:54 +0200
    message:
      Fix for Bug#37853
          Test "funcs_1.processlist_val_ps" fails in various ways
      + corrections of logic in poll routines
      + minor improvements

This was a deadlock between LOCK TABLES/CREATE DATABASE in one connection
and DROP DATABASE in another. It only happened if the table locked by 
LOCK TABLES was in the database to be dropped. The deadlock is similar
to the one in Bug#48940, but with LOCK TABLES instead of an active
transaction.

The order of events needed to trigger the deadlock was:
1) Connection 1 locks table db1.t1 using LOCK TABLES. It will now
have a metadata lock on the table name.
2) Connection 2 issues DROP DATABASE db1. This will wait inside
the MDL subsystem for the lock on db1.t1 to go away. While waiting, it
will hold the LOCK_mysql_create_db mutex.
3) Connection 1 issues CREATE DATABASE (database name irrelevant).
This will hang trying to lock the same mutex. Since this is the connection
holding the metadata lock blocking Connection 2, we have a deadlock.

This deadlock would also happen for earlier trees without MDL, but 
there DROP DATABASE would wait for a table to be removed from the
table definition cache.

This patch fixes the problem by prohibiting CREATE DATABASE in LOCK TABLES
mode. In the example above, this prevents Connection 1 from hanging trying
to get the LOCK_mysql_create_db mutex. Note that other commands that use
LOCK_mysql_create_db (ALTER/DROP DATABASE) are already prohibited in 
LOCK TABLES mode.

Incompatible change: CREATE DATABASE is now disallowed in LOCK TABLES mode.

Test case added to schema.test.


mysql-test/t/drop.test:
  Updates the test for Bug#21216 by swapping the order of CREATE DATABASE
  and LOCK TABLES. This is now needed as CREATE DATABASE is prohibited in
  LOCK TABLES mode.
mysql-test/t/schema.test:
  Test case for Bug#49988 added.
  Also fixes a problem with the test for Bug#48940 where the result 
  would differ for embedded server.

Bug#45523 "Objects of class base_ilist should not be copyable".
               
Suppress the compiler-generated public copy constructor
and assignment operator of class base_ilist; instead, implement
move_elements_to() function which transfers ownership of elements
from one list to another.

                   INFILE".

Attempts to execute an INSERT statement for a MEMORY table which invoked
a trigger or called a stored function which tried to perform LOW_PRIORITY
update on the table being inserted into, resulted in debug servers aborting
due to an assertion failure. On non-debug servers such INSERTs failed with
"Can't update table t1 in stored function/trigger because it is already used
by statement which invoked this stored function/trigger" as expected.

The problem was that in the above scenario TL_WRITE_CONCURRENT_INSERT
is converted to TL_WRITE inside the thr_lock() function since the MEMORY
engine does not support concurrent inserts. This triggered an assertion
which assumed that for the same table, one thread always requests locks with
higher thr_lock_type value first. When TL_WRITE_CONCURRENT_INSERT is
upgraded to TL_WRITE after the locks have been sorted, this is no longer true.
In this case, TL_WRITE was requested after acquiring a TL_WRITE_LOW_PRIORITY
lock on the table, triggering the assert.

This fix solves the problem by adjusting this assert to take this
scenario into account.

An alternative approach to change handler::store_locks() methods for all engines
which do not support concurrent inserts in such way that
TL_WRITE_CONCURRENT_INSERT is upgraded to TL_WRITE there instead, 
was considered too intrusive.

Commit on behalf of Dmitry Lenev.


mysql-test/r/lock.result:
  Added simplified test for bug #48538 "Assertion in thr_lock() on LOAD
  DATA CONCURRENT INFILE".
mysql-test/t/lock.test:
  Added simplified test for bug #48538 "Assertion in thr_lock() on LOAD
  DATA CONCURRENT INFILE".
mysys/thr_lock.c:
  Adjusted assertion to account for situation when
  TL_WRITE_CONCURRENT_INSERT is converted to TL_WRITE inside of the
  thr_lock() function because the engine of the table being locked 
  does not support concurrent inserts.
  This scenario breaks assumption that for the same table one thread
  always requests locks with higher thr_lock_type value first, since
  TL_WRITE on the table (converted from TL_WRITE_CONCURRENT_INSERT)
  can be requested after acquiring a TL_WRITE_LOW_PRIORITY lock on the table.
  Note that it is still safe to grant a new lock without extra checks and
  waiting in such situation since TL_WRITE has the same compatibility
  rules as TL_WRITE_LOW_PRIORITY (their only difference is priority).

This change is supposed to reduce number of ER_LOCK_DEADLOCK
errors which occur when multi-statement transaction encounters
conflicting metadata lock in cases when waiting is possible.

The idea is not to fail ER_LOCK_DEADLOCK error immediately when
we encounter conflicting metadata lock. Instead we release all
metadata locks acquired by current statement and start to wait
until conflicting lock go away. To avoid deadlocks we use simple
empiric which aborts waiting with ER_LOCK_DEADLOCK error if it
turns out that somebody is waiting for metadata locks owned by
this transaction.

This patch also fixes bug #46273 "MySQL 5.4.4 new MDL: Bug#989
is not fully fixed in case of ALTER".

The bug was that concurrent execution of UPDATE or MULTI-UPDATE
statement as a part of multi-statement transaction that already
has used table being updated and ALTER TABLE statement might have
resulted of loss of isolation between this transaction and ALTER
TABLE statement, which manifested itself as changes performed by
ALTER TABLE becoming visible in transaction and wrong binary log
order as a consequence.

This problem occurred when UPDATE or MULTI-UPDATE's wait in
mysql_lock_tables() call was aborted due to metadata lock
upgrade performed by concurrent ALTER TABLE. After such abort all
metadata locks held by transaction were released but transaction
silently continued to be executed as if nothing has happened.

We solve this problem by changing our code not to release all
locks in such case. Instead we release only locks which were
acquired by current statement and then try to reacquire them
by restarting open/lock tables process. We piggyback on simple
deadlock detector implementation since this change has to be
done anyway for it.

mysql-test/include/handler.inc:
  After introduction of basic deadlock detector for metadata locks
  it became necessary to change parts of test for HANDLER statements
  which covered some of scenarios in which ER_LOCK_DEADLOCK error
  was detected in absence of real deadlock (with new deadlock detector
  this no longer happens).
  Also adjusted test to the fact that HANDLER READ for the table no
  longer will be blocked by ALTER TABLE for the same table which awaits
  for metadata lock upgrade (this is due to removal of mysql_lock_abort()
  from wait_while_table_is_used()).
mysql-test/r/handler_innodb.result:
  After introduction of basic deadlock detector for metadata locks
  it became necessary to change parts of test for HANDLER statements
  which covered some of scenarios in which ER_LOCK_DEADLOCK error
  was detected in absence of real deadlock (with new deadlock detector
  this no longer happens).
  Also adjusted test to the fact that HANDLER READ for the table no
  longer will be blocked by ALTER TABLE for the same table which awaits
  for metadata lock upgrade (this is due to removal of mysql_lock_abort()
  from wait_while_table_is_used()).
mysql-test/r/handler_myisam.result:
  After introduction of basic deadlock detector for metadata locks
  it became necessary to change parts of test for HANDLER statements
  which covered some of scenarios in which ER_LOCK_DEADLOCK error
  was detected in absence of real deadlock (with new deadlock detector
  this no longer happens).
  Also adjusted test to the fact that HANDLER READ for the table no
  longer will be blocked by ALTER TABLE for the same table which awaits
  for metadata lock upgrade (this is due to removal of mysql_lock_abort()
  from wait_while_table_is_used()).
mysql-test/r/mdl_sync.result:
  Added test coverage for basic deadlock detection in metadata
  locking subsystem and for bug #46273 "MySQL 5.4.4 new MDL:
  Bug#989 is not fully fixed in case of ALTER".
mysql-test/r/sp-lock.result:
  Adjusted test coverage for metadata locking for stored routines
  since after introduction of basic deadlock detector for metadata
  locks number of scenarios in which ER_LOCK_DEADLOCK error in
  absence of deadlock has decreased.
mysql-test/t/mdl_sync.test:
  Added test coverage for basic deadlock detection in metadata
  locking subsystem and for bug #46273 "MySQL 5.4.4 new MDL:
  Bug#989 is not fully fixed in case of ALTER".
mysql-test/t/sp-lock.test:
  Adjusted test coverage for metadata locking for stored routines
  since after introduction of basic deadlock detector for metadata
  locks number of scenarios in which ER_LOCK_DEADLOCK error in
  absence of deadlock has decreased.
sql/log_event_old.cc:
  close_tables_for_reopen() now takes one more argument which
  specifies at which point it should stop releasing metadata
  locks acquired by this connection.
sql/mdl.cc:
  Changed metadata locking subsystem to support basic deadlock detection
  with a help of the following simple empiric -- we assume that there is
  a deadlock if there is a connection which has to wait for a metadata
  lock which is currently acquired by some connection which is itself
  waiting to be able to acquire some shared metadata lock.
  
  To implement this change:
  - Added MDL_context::can_wait_lead_to_deadlock()/_impl() methods
    which allow to find out if there is someone waiting for metadata
    lock which is held by the connection and therefore deadlocks are
    possible if this connection is going to wait for some metadata lock.
    To do this added version of MDL_ticket::has_pending_conflicting_lock()
    method which assumes that its caller already owns LOCK_mdl mutex.
  - Changed MDL_context::wait_for_locks() to use one of the above methods
    to check if somebody is waiting for metadata lock owned by this
    context (and therefore deadlock is possible) and emit ER_LOCK_DEADLOCK
    error in this case. Also now we mark context of connections waiting
    inside of this method by setting MDL_context::m_is_waiting_in_mdl
    member. Thanks to this such connection could be waken up if some
    other connection starts waiting for one of its metadata locks and
    so a deadlock can occur.
  - Adjusted notify_shared_lock() to wake up connections which wait inside
    MDL_context::wait_for_locks() while holding shared metadata lock.
  - Changed MDL_ticket::upgrade_shared_lock_to_exclusive() to add
    temporary ticket for exclusive lock to MDL_lock::waiting queue, so
    request for metadata lock upgrade can be properly detected by our
    empiric.
    Also now this method invokes a callback which forces transactions
    holding shared metadata lock on the table to call MDL_context::
    can_wait_lead_to_deadlock() method even if they don't need any new
    metadata locks. Thanks to this such transactions can detect deadlocks/
    livelocks between MDL and table-level locks.
  
  Also reduced timeouts between calls to notify_shared_lock()
  in MDL_ticket::upgrade_shared_lock_to_exclusive() and
  MDL_context::acquire_exclusive_locks(). This was necessary
  to get rid of call to mysql_lock_abort() in wait_while_table_is_used().
  (Now we instead rely on notify_shared_lock() timely calling
  mysql_lock_abort_for_thread() for the table on which lock
  is being upgraded/acquired).
sql/mdl.h:
  - Added a version of MDL_ticket::has_pending_conflicting_lock() method
    to be used in situations when caller already has acquired LOCK_mdl
    mutex.
  - Added MDL_context::can_wait_lead_to_deadlock()/_impl() methods
    which allow to find out if there is someone waiting for metadata lock
    which is held by this connection and thus deadlocks are possible if
    this connections will start waiting for some metadata lock.
  - Added MDL_context::m_is_waiting_in_mdl member to mark connections
    waiting in MDL_context::wait_for_locks() method of metadata locking
    subsystem. Added getter method for this private member to make it
    accessible in notify_shared_lock() auxiliary so we can wake-up such
    connections if they hold shared metadata locks.
  - Finally, added mysql_abort_transactions_with_shared_lock() callback
    to be able force transactions which don't need any new metadata
    locks still call MDL_context::can_wait_lead_to_deadlock() and detect
    some of deadlocks between metadata locks and table-level locks.
sql/mysql_priv.h:
  close_tables_for_reopen() now takes one more argument which
  specifies at which point it should stop releasing metadata
  locks acquired by this connection.
sql/sql_base.cc:
  Changed approach to metadata locking for multi-statement transactions.
  We no longer fail ER_LOCK_DEADLOCK error immediately when we encounter
  conflicting metadata lock. Instead we release all metadata locks
  acquired by current statement and start to wait until conflicting
  locks to go away by calling MDL_context::wait_for_locks() method.
  To avoid deadlocks the latter implements simple empiric which aborts
  waiting with ER_LOCK_DEADLOCK error if it turns out that somebody
  is waiting for metadata locks owned by this transaction.
  
  To implement the change described above:
  - Introduced Open_table_context::m_start_of_statement_svp member to
    store state of metadata locks at the start of the statement.
  - Changed Open_table_context::request_backoff_action() not to
    fail with ER_LOCK_DEADLOCK immediately if back-off is requested
    due to conflicting metadata lock.
  - Added new argument for close_tables_for_reopen() procedure which
    allows to specify subset of metadata locks to be released.
  - Changed open_tables() not to release all metadata locks acquired
    by current transaction when metadata lock conflict is discovered.
    Instead we release only locks acquired by current statement.
  - Changed open_ltable() and open_and_lock_tables_derived() not to emit
    ER_LOCK_DEADLOCK error when mysql_lock_tables() is aborted in
    multi-statement transaction when somebody tries to acquire exclusive
    metadata lock on the table. Instead we release metadata locks acquired
    by current statement and try to wait until they can be re-acquired.
  - Adjusted tdc_wait_for_old_versions() to check if there is someone
    waiting for one of metadata locks held by this connection and run
    deadlock detection in order to avoid deadlocks in some
    situations.
  - Added mysql_abort_transactions_with_shared_lock() callback which
    allows to force transactions holding shared metadata lock on the
    table to call MDL_context::can_wait_lead_to_deadlock() even if they
    don't need any new metadata locks so they can detect potential
    deadlocks between metadata locking subsystem and table-level locks.
  - Adjusted wait_while_table_is_used() not to set TABLE::version to
    0 as it is now done only when necessary by the above-mentioned
    callback. Also removed unnecessary call to mysql_lock_abort().
    Instead we rely on code performing metadata lock upgrade aborting
    waits on the table-level lock for this table by calling
    mysql_lock_abort_for_thread() (invoked by
    mysql_notify_thread_having_shared_lock()). In future this should
    allow to reduce number of scenarios in which we produce
    ER_LOCK_DEADLOCK error even though no real deadlock exists.
sql/sql_class.h:
  Introduced Open_table_context::m_start_of_statement_svp member to
  store state of metadata locks at the start of the statement.
  Replaced Open_table_context::m_can_deadlock member with m_has_locks
  member to reflect the fact that we no longer unconditionally emit
  ER_LOCK_DEADLOCK error for transaction having some metadata locks
  when conflicting metadata lock is discovered.
sql/sql_insert.cc:
  close_tables_for_reopen() now takes one more argument which
  specifies at which point it should stop releasing metadata
  locks acquired by this connection.
sql/sql_plist.h:
  Made I_P_List_iterator<T, B> usable with const lists.
sql/sql_show.cc:
  close_tables_for_reopen() now takes one more argument which
  specifies at which point it should stop releasing metadata
  locks acquired by this connection.
sql/sql_update.cc:
  Changed UPDATE and MULTI-UPDATE code not to release all metadata
  locks when calls to mysql_lock_tables() are aborted. Instead we
  release only locks which are acquired by this statement and then
  try to reacquire them by calling open_tables(). This solves
  bug #46273 "MySQL 5.4.4 new MDL: Bug#989 is not fully fixed in
  case of ALTER".

3655 Jon Olav Hauglid   2009-10-19
Bug #30977 Concurrent statement using stored function and DROP FUNCTION 
           breaks SBR
Bug #48246 assert in close_thread_table

Implement a fix for:
Bug #41804 purge stored procedure cache causes mysterious hang for many
           minutes
Bug #49972 Crash in prepared statements

The problem was that concurrent execution of DML statements that
use stored functions and DDL statements that drop/modify the same
function might result in incorrect binary log in statement (and
mixed) mode and therefore break replication.

This patch fixes the problem by introducing metadata locking for
stored procedures and functions. This is similar to what is done
in Bug#25144 for views. Procedures and functions now are
locked using metadata locks until the transaction is either
committed or rolled back. This prevents other statements from
modifying the procedure/function while it is being executed. This
provides commit ordering - guaranteeing serializability across
multiple transactions and thus fixes the reported binlog problem.

Note that we do not take locks for top-level CALLs. This means
that procedures called directly are not protected from changes by
simultaneous DDL operations so they are executed at the state they
had at the time of the CALL. By not taking locks for top-level
CALLs, we still allow transactions to be started inside
procedures.

This patch also changes stored procedure cache invalidation.
Upon a change of cache version, we no longer invalidate the entire
cache, but only those routines which we use, only when a statement
is executed that uses them.

This patch also changes the logic of prepared statement validation.
A stored procedure used by a prepared statement is now validated
only once a metadata lock has been acquired. A version mismatch
causes a flush of the obsolete routine from the cache and
statement reprepare.
Incompatible changes:
1) ER_LOCK_DEADLOCK is reported for a transaction trying to access
   a procedure/function that is locked by a DDL operation in
   another connection.

2) Procedure/function DDL operations are now prohibited in LOCK
   TABLES mode as exclusive locks must be taken all at once and
   LOCK TABLES provides no way to specifiy procedures/functions to
   be locked.

Test cases have been added to sp-lock.test and rpl_sp.test.

Work on this bug has very much been a team effort and this patch
includes and is based on contributions from Davi Arnaut, Dmitry
Lenev, Magne Mæhre and Konstantin Osipov.


mysql-test/r/ps_ddl.result:
  Update results (Bug#30977).
mysql-test/r/ps_ddl1.result:
  Update results (Bug#30977).
mysql-test/r/sp-error.result:
  Update results (Bug#30977).
mysql-test/r/sp-lock.result:
  Update results (Bug#30977).
mysql-test/suite/rpl/r/rpl_sp.result:
  Update results (Bug#30977).
mysql-test/suite/rpl/t/rpl_sp.test:
  Add a test case for Bug#30977.
mysql-test/t/ps_ddl.test:
  Update comments. We no longer re-prepare a prepared statement
  when a stored procedure used in top-level CALL is changed.
mysql-test/t/ps_ddl1.test:
  Modifying stored procedure p1 no longer invalidates prepared
  statement "call p1" -- we can re-use the prepared statement
  without invalidation.
mysql-test/t/sp-error.test:
  Use a constant for an error value.
mysql-test/t/sp-lock.test:
  Add test coverage for Bug#30977.
sql/lock.cc:
  Implement lock_routine_name() - a way to acquire an 
  exclusive metadata lock (ex- name-lock) on 
  stored procedure/function.
sql/sp.cc:
  Change semantics of sp_cache_routine() -- now it has an option
  to make sure that the routine that is cached is up to date (has
  the latest sp cache version).
  
  Add sp_cache_invalidate() to sp_drop_routine(), where it was
  missing (a bug!).
  
  Acquire metadata locks for SP DDL (ALTER/CREATE/DROP). This is
  the core of the fix for Bug#30977.
  
  Since caching and cache invalidation scheme was changed, make 
  sure we don't invalidate the SP cache in the middle of a stored
  routine execution. At the same time, make sure we don't access
  stale data due to lack of invalidation. 
  For that, change ALTER FUNCTION/PROCEDURE to not use the cache,
  and SHOW PROCEDURE CODE/SHOW CREATE PROCEDURE/FUNCTION to always
  read an up to date version of the routine from the cache.
sql/sp.h:
  Add a helper wrapper around sp_cache_routine().
sql/sp_cache.cc:
  Implement new sp_cache_version() and sp_cache_flush_obsolete().
  Now we flush stale routines individually, rather than all at once.
sql/sp_cache.h:
  Update signatures of sp_cache_version() and sp_cache_flush_obsolete().
sql/sp_head.cc:
  Add a default initialization of sp_head::m_sp_cache_version.
  Remove a redundant sp_head::create().
sql/sp_head.h:
  Add m_sp_cache_version to sp_head class - we now 
  keep track of every routine in the stored procedure cache, rather than
  of the entire cache.
sql/sql_base.cc:
  Implement prelocking for stored routines. Validate stored
  routines after they were locked.
  Flush obsolete routines upon next access, one by one, not all at once
  (Bug#41804).
  Style fixes.
sql/sql_class.h:
  Rename a Open_table_context method.
sql/sql_parse.cc:
  Make sure stored procedures DDL commits the active transaction 
  (issues an implicit commit before and after).
  Remove sp_head::create(), a pure redundancy.
  Move the semantical check during alter routine inside sp_update_routine() code in order to:
  - avoid using SP cache during update, it may be obsolete.
  - speed up and simplify the update procedure.
  
  Remove sp_cache_flush_obsolete() calls, we no longer flush the entire
  cache, ever, stale routines are flushed before next use, one at a time.
sql/sql_prepare.cc:
  Move routine metadata validation to open_and_process_routine().
  Fix Bug#49972 (don't swap flags at reprepare).
  Reset Sroutine_hash_entries in reinit_stmt_before_use().
  Remove SP cache invalidation, it's now done by open_tables().
sql/sql_show.cc:
  Fix a warning: remove an unused label.
sql/sql_table.cc:
  Reset mdl_request.ticket for tickets acquired for routines inlined
  through a view, in CHECK TABLE statement, to satisfy an MDL assert.
sql/sql_update.cc:
  Move the cleanup of "translation items" to close_tables_for_reopen(),
  since it's needed in all cases when we back off, not just
  the back-off in multi-update. This fixes a bug when the server
  would crash on attempt to back off when opening tables
  for a statement that uses information_schema tables.

"HANDLER statements within a transaction might lead to deadlocks".
Introduce a notion of a sentinel to MDL_context. A sentinel
is a ticket that separates all tickets in the context into two
groups: before and after it. Currently we can have (and need) only
one designated sentinel -- it separates all locks taken by LOCK
TABLE or HANDLER statement, which must survive COMMIT and ROLLBACK
and all other locks, which must be released at COMMIT or ROLLBACK.
The tricky part is maintaining the sentinel up to date when
someone release its corresponding ticket. This can happen, e.g.
if someone issues DROP TABLE under LOCK TABLES (generally,
see all calls to release_all_locks_for_name()).
MDL_context::release_ticket() is modified to take care of it.

******
A fix and a test case for Bug#46224 "HANDLER statements within a
transaction might lead to deadlocks".

An attempt to mix HANDLER SQL statements, which are transaction-
agnostic, an open multi-statement transaction,
and DDL against the involved tables (in a concurrent connection) 
could lead to a deadlock. The deadlock would occur when
HANDLER OPEN or HANDLER READ would have to wait on a conflicting
metadata lock. If the connection that issued HANDLER statement
also had other metadata locks (say, acquired in scope of a 
transaction), a classical deadlock situation of mutual wait
could occur.

Incompatible change: entering LOCK TABLES mode automatically
closes all open HANDLERs in the current connection.

Incompatible change: previously an attempt to wait on a lock
in a connection that has an open HANDLER statement could wait
indefinitely/deadlock. After this patch, an error ER_LOCK_DEADLOCK
is produced.

The idea of the fix is to merge thd->handler_mdl_context
with the main mdl_context of the connection, used for transactional
locks. This makes deadlock detection possible, since all waits
with locks are "visible" and available to analysis in a single
MDL context of the connection.

Since HANDLER locks and transactional locks have a different life
cycle -- HANDLERs are explicitly open and closed, and so
are HANDLER locks, explicitly acquired and released, whereas
transactional locks "accumulate" till the end of a transaction
and are released only with COMMIT, ROLLBACK and ROLLBACK TO SAVEPOINT,
a concept of "sentinel" was introduced to MDL_context.
All locks, HANDLER and others, reside in the same linked list.
However, a selected element of the list separates locks with
different life cycle. HANDLER locks always reside at the
end of the list, after the sentinel. Transactional locks are
prepended to the beginning of the list, before the sentinel.
Thus, ROLLBACK, COMMIT or ROLLBACK TO SAVEPOINT, only
release those locks that reside before the sentinel. HANDLER locks
must be released explicitly as part of HANDLER CLOSE statement,
or an implicit close. 
The same approach with sentinel
is also employed for LOCK TABLES locks. Since HANDLER and LOCK TABLES
statement has never worked together, the implementation is
made simple and only maintains one sentinel, which is used either
for HANDLER locks, or for LOCK TABLES locks.


mysql-test/include/handler.inc:
  Add test coverage for Bug#46224 "HANDLER statements within a
  transaction might lead to deadlocks".
  Extended HANDLER coverage to cover a mix of HANDLER, transactions
  and DDL statements.
mysql-test/r/handler_innodb.result:
  Update results (Bug#46224).
mysql-test/r/handler_myisam.result:
  Update results (Bug#46224).
sql/lock.cc:
  Remove thd->some_tables_deleted, it's never used.
sql/log_event.cc:
  No need to check for thd->locked_tables_mode, 
  it's done inside release_transactional_locks().
sql/mdl.cc:
  Implement the concept of HANDLER and LOCK TABLES "sentinel".
  Implement a method to clone an acquired ticket.
  Do not return tickets beyond the sentinel when acquiring
  locks, create a copy.
  Remove methods to merge and backup MDL_context, they are now
  not used (Hurra!). This opens a path to a proper constructor
  and destructor of class MDL_context (to be done in a separate
  patch).
  Modify find_ticket() to provide information about where
  the ticket position is with regard to the sentinel.
sql/mdl.h:
  Add declarations necessary for the implementation of the concept
  of "sentinel", a dedicated ticket separating transactional and
  non-transactional locks.
sql/mysql_priv.h:
  Add mark_tmp_table_for_reuse() declaration, 
  a function to "close" a single session (temporary) table.
sql/sql_base.cc:
  Remove thd->some_tables_deleted.
  Modify deadlock-prevention asserts and deadlock detection
  heuristics to take into account that from now on HANDLER locks
  reside in the same locking context.
  Add broadcast_refresh() to mysql_notify_thread_having_shared_lock():
  this is necessary for the case when a thread having a shared lock
  is asleep in tdc_wait_for_old_versions(). This situation is only
  possible with HANDLER t1 OPEN; FLUSH TABLE (since all over code paths
  that lead to tdc_wait_for_old_versions() always have an
  empty MDL_context). Previously the server would simply deadlock
  in this situation.
sql/sql_class.cc:
  Remove now unused member "THD::some_tables_deleted". 
  Move mysql_ha_cleanup() a few lines above in THD::cleanup() 
  to make sure that all handlers are closed when it's time to 
  destroy the MDL_context of this connection.
  Remove handler_mdl_context and handler_tables.
sql/sql_class.h:
  Remove THD::handler_tables, THD::handler_mdl_context,
  THD::some_tables_deleted.
sql/sql_handler.cc:
  Remove thd->handler_tables.
  Remove thd->handler_mdl_context.
  Rewrite mysql_ha_open() to have no special provision for MERGE
  tables, now that we don't have to manipulate with thd->handler_tables
  it's easy to do.
  Remove dead code.
  Fix a bug in mysql_ha_flush() when we would always flush
  a temporary HANDLER when mysql_ha_flush() is called (actually
  mysql_ha_flush() never needs to flush temporary tables).
sql/sql_insert.cc:
  Update a comment, no more thd->some_tables_deleted.
sql/sql_parse.cc:
  Implement an incompatible change: entering LOCK TABLES closes
  active HANDLERs, if any.
  Now that we have a sentinel, we don't need to check
  for thd->locked_tables_mode when releasing metadata locks in
  COMMIT/ROLLBACK.
sql/sql_plist.h:
  Add new (now necessary) methods to the list class.
sql/sql_prepare.cc:
  Make sure we don't release HANDLER locks when rollback to a
  savepoint, set to not keep locks taken at PREPARE.
sql/sql_servers.cc:
  Update to a new signature of MDL_context::release_all_locks().
sql/sql_table.cc:
  Remove thd->some_tables_deleted.
sql/transaction.cc:
  Add comments. 
  Make sure rollback to (MDL) savepoint works under LOCK TABLES and
  with HANDLER tables.

------------------------------------------------------------
revno: 2617.14.26
committer: Vladislav Vaintroub <vvaintroub@mysql.com>
branch nick: mysql-6.0-wtf
timestamp: Wed 2008-11-05 11:19:19 +0100
message:
   CMakeLists.txt files cleanup.
  
  - remove SAFEMALLOC and SAFE_MUTEX definitions that were
  present in *each* CMakeLists.txt. Instead, put them into top level
  MakeLists.txt, but disable on Windows, because
  
  a) SAFEMALLOC does not add any functionality that is not already
  present in Debug C runtime ( and 2 safe malloc one on top of the other
  only unnecessarily slows down the server)
  
  b)SAFE_MUTEX does not work on Windows  and have been
  explicitely  disabled on Windows with #undef previously.  Fortunately,
  ntdll does  pretty good  job identifying l problems with  CRITICAL_SECTIONs.
  (DebugBreak()s on using uninited critical section, unlocking unowned
  critical section)
  
  -Remove occationally used -D_DEBUG (added by compiler
  anyway)
  
  -Remove MAP file generation, it became  obsolete .
  There are many ways to get callstack  of a crash now, with stacktrace in 
  error log , minidump etc

If the handler (or delayed insert) thread failed to lock a table due
to being killed, the "dead" flag was used to notify the connection thread
of this failure. However, with the changes introduced by Bug#45949, 
the handler thread will no longer try to lock the table if it was killed.
This meant that the "dead" flag would not be set, and the connection
thread would not notice that the handler thread had failed.

This could happen with concurrent INSERT DELAYED and FLUSH TABLES.
FLUSH TABLES would kill any active INSERT DELAYED that had opened any
table(s) to be flushed. This could cause the INSERT DELAYED connection
thread to be stuck waiting for the handler thread to lock its table,
while the handler thread would be looping, trying to get the connection
thread to notice the error.

The root of the problem was that the handler thread had both the "dead"
flag and "thd->killed" to indicate that it had been killed. Most places
both were set, but some only set "thd->killed". And 
Delayed_insert::get_local_table() only checked "dead" while waiting for
the table to be locked.

This patch removes the "dead" variable and replaces its usage with
"thd->killed", thereby resolving the issue.

The reason for the deadlock was an improper exit from
MDL_context::wait_for_locks() which caused mysys_var->current_mutex to remain
LOCK_mdl even though LOCK_mdl was no longer held by that connection. 

This could for example lead to a deadlock in the following way:
1) INSERT DELAYED tries to open a table but fails, and trying to recover it
calls wait_for_locks().
2) Due to a pending exclusive request, wait_for_locks() fails and exits without
resetting mysys_var->current_mutex for the delayed insert handler thread. So it
continues to point to LOCK_mdl.
3) The handler thread manages to open a table.
4) A different connection takes LOCK_open and tries to take LOCK_mdl.
5) FLUSH TABLES from a third connection notices that the handler thread has a
table open, and tries to kill it. This involves locking mysys_var->current_mutex
while having LOCK_open locked. Since current_mutex mistakenly points to LOCK_mdl,
we have a deadlock.

This patch makes sure MDL_EXIT_COND() is called before exiting wait_for_locks().
This clears mysys->current_mutex which resolves the issue. 

An assert is added to recover_from_failed_open_table_attempt() after
wait_for_locks() is called, to check that current_mutex is indeed reset.
With this assert in place, existing tests in (e.g.) mdl_sync.test will fail
without this patch.

the fix for Bug#37148, since it is null-merged into 6.0.

This deadlock would occur between two connections A and B if statements
where executed in the following way:
1) Connection A executes a DML statement against table s1.t1 with
autocommit off. This causes a shared metadata lock on s1.t1 to be 
acquired. (With autocommit on, the metadata lock will be dropped once
the statment completes and the deadlock will not occour.)
2) Connection B tries to DROP DATABASE s1. This will block against the
metadata lock connection A holds on s1.t1. While blocking, connection B
will hold the LOCK_mysql_create_db mutex.
3) Connection A tries to ALTER DATABASE s1. This will block when trying
to get LOCK_mysql_create_db mutex held by connection B.
4) Deadlock between DROP DATABASE and ALTER DATABASE (which has autocommit
off).

If Connection A used an explicitly started transaction rather than having
autocommit off, this deadlock did not happen as ALTER DATABASE is 
disallowed inside transactions.

This patch fixes the problem by changing ALTER DATABASE to cause an
implicit commit before executing. This will cause the metadata 
lock on s1.t1 to be dropped, allowing DROP DATABASE to proceed. 
This will in turn cause the LOCK_mysql_create_db mutex to be unlocked, 
allowing ALTER DATABASE to proceed.

Note that SQL commands other than ALTER DATABASE that also use 
LOCK_mysql_create_db, already cause an implicit commit. 

Incompatible change: ALTER DATABASE (and its synonym ALTER SCHEMA)
now cause an implicit commit. This must be reflected in the 
documentation.

Test case added to schema.test.


sql/sql_parse.cc:
  Added CF_AUTO_COMMIT_TRANS to SQLCOM_ALTER_DB.
  
  Removed thd->active_transaction() checks from SQLCOM_DROP_DB, 
  SQLCOM_ALTER_DB_UPGRADE and SQLCOM_ALTER_DB as these statements
  cause an implicit commit.

------------------------------------------------------------
 2599.161.3 Ingo Struewing      2009-07-21
 Bug#20667 - Truncate table fails for a write locked table

 TRUNCATE TABLE was not allowed under LOCK TABLES.

 The patch removes this restriction. mysql_truncate()
 does now handle that case.


mysql-test/r/merge.result:
  Bug#20667 - Truncate table fails for a write locked table
  Updated test result.
mysql-test/r/truncate.result:
  Bug#20667 - Truncate table fails for a write locked table
  Updated test result.
mysql-test/r/truncate_coverage.result:
  Bug#20667 - Truncate table fails for a write locked table
  New test result.
mysql-test/t/merge.test:
  Bug#20667 - Truncate table fails for a write locked table
  Updated test case due to now working TRUNCATE under LOCK TABLES.
  Added some SELECTs to show that child tables are truncated.
mysql-test/t/truncate.test:
  Bug#20667 - Truncate table fails for a write locked table
  Added test cases for TRUNCATE under LOCK TABLE.
mysql-test/t/truncate_coverage.test:
  Bug#20667 - Truncate table fails for a write locked table
  New test file. Coverage tests for TRUNCATE.
sql/sql_delete.cc:
  Bug#20667 - Truncate table fails for a write locked table
  Added branches for thd->locked_tables_mode.
sql/sql_parse.cc:
  Bug#20667 - Truncate table fails for a write locked table
  Deleted rejection of TRUNCATE in case of LOCK TABLES.

-----------------------------------------------------------
2630.28.28 Magne Mahre  2008-12-05
Bug #38661 'all threads hang in "opening tables" or "waiting for table"
            and cpu is at 100%'
                      
Concurrent execution of FLUSH TABLES statement and at least two statements
using the same table might have led to live-lock which caused all three
connections to stall and hog 100% of CPU.
        
tdc_wait_for_old_versions() wrongly assumed that there cannot be a share
with an old version and no used TABLE instances and thus was failing to
perform wait in situation when such old share was cached in MDL subsystem
thanks to a still active metadata lock on the table. So it might have
happened that two or more connections simultaneously executing statements
which involve table being flushed managed to prevent each other from
waiting in this function by keeping shared metadata lock on the table 
constantly active (i.e. one of the statements managed to take/hold this
lock while other statements were calling tdc_wait_for_old_versions()).
Thus they were forcing each other to loop infinitely in open_tables() - 
close_thread_tables_for_reopen() - tdc_wait_for_old_versions() cycle
causing CPU hogging.
        
This patch fixes this problem by removing this false assumption from
tdc_wait_for_old_versions().
 
Note that the problem is specific only for server versions >= 6.0.
        
No test case is submitted for this test, as the test infrastructure
hasn't got the necessary primitives to test the behaviour.  The
manifestation is that throughput will decrease to a low level
(possibly 0) after some time, and stay at that level. Several
transactions will not complete. 
        
Manual testing can be done by running the code submitted by Shane 
Bester attached to the bug report.  If the bug persists, the 
transaction thruput will almost immediately drop to near zero 
(shown as the transaction count output from the test program staying 
on a close to constant value, instead of increasing rapidly).

-----------------------------------------------------------
2497.392.1 Michael Widenius	2008-08-19
Fixes for Bug #38016 Maria: trying to access freed memory when
committing a transaction.
Don't write out states if they haven't changed.


sql/sql_table.cc:
  Call extra(HA_EXTRA_PREPARE_FOR_RENAME) before renaming a table.

----------------------------------------------------
2736.2.10 Michael Widenius	2008-10-22
Fix for bug#39395 Maria: ma_extra.c:286: maria_extra: 
Assertion `share->reopen == 1' failed


sql/sql_base.cc:
  Race condition in wait_while_table_is_used() where a table used 
  by another connection could be forced closed, but there was no protection against the other thread re-opening the table and trying to lock it 
  again before the table was name locked by original thread.

                  
An error occuring in the execution of a stored procedure, called
from do_select is masked, since the error condition is not
propagated back to the caller (join->conds->val_int() returns
a result value, and not an error code)
                  
An explicit check was added to see if the thd error code has been
set, and if so, the loop status is set to the error state.

Backport from 6.0-codebase (revid: 2617.68.31)

          (diagnostics_area)
      
Execution of CREATE TABLE ... SELECT statement was not atomic in
the sense that concurrent statements trying to affect its target
table might have sneaked in between the moment when the table was
created and moment when it was filled according to SELECT clause.
This resulted in inconsistent binary log, unexpected target table
contents. In cases when concurrent statement was a DDL statement
CREATE TABLE ... SELECT might have failed with ER_CANT_LOCK error.
      
In more detail:
Due to premature metadata lock downgrade which occured after CREATE
TABLE SELECT statement created table but before it managed to obtain
table-level lock on it other statements were allowed to open, lock
and change target table in the middle of CREATE TABLE SELECT
execution. This also meant that it was possible that CREATE TABLE
SELECT would wait in mysql_lock_tables() when it was called for newly
created table and that this wait could have been aborted by concurrent
DDL. The latter led to execution of unexpected branch of code and
CREATE TABLE SELECT ending with ER_CANT_LOCK error.
      
The premature downgrade occured because open_table(), which was called
for newly created table, decided that it is OK to downgrade metadata
lock from exclusive to shared since table exists, even although it
was not acquired within this call.
      
This fix ensures that open_table() does not downgrade metadata lock
if it is not acquired during its current invocation.
      
Testing:
The bug is exposed in a race condition, and is thus difficult to
expose in a standard mysql-test-run test case.  Instead, a stress
test using the Random Query Generator (https://launchpad.net/randgen)
will trip the problem occasionally.
      
   % perl  runall.pl \
            --basedir=<build dir> \
             --mysqld=--table-lock-wait-timeout=5 \
             --mysqld=--skip-safemalloc \
             --grammar=conf/maria_bulk_insert.yy \
             --reporters=ErrorLog,Backtrace,WinPackage \
             --mysqld=--log-output=file  \
             --queries=100000 \
             --threads=10 \
             --engine=myisam
      
Note: You will need a debug build to expose the bug
      
When the bug is tripped, the server will abort and dump core.


Backport from 6.0-codebase   (revid: 2617.53.4)

Postfix for Bug#48210 FLUSH TABLES WITH READ LOCK deadlocks
                      against concurrent CREATE PROCEDURE

Rewrote the second test to use DROP PROCEDURE instead of 
CREATE USER as CREATE USER does not work with embedded server.