Implementation of simple deadlock detection for metadata locks.

This change is supposed to reduce number of ER_LOCK_DEADLOCK
errors which occur when multi-statement transaction encounters
conflicting metadata lock in cases when waiting is possible.

The idea is not to fail ER_LOCK_DEADLOCK error immediately when
we encounter conflicting metadata lock. Instead we release all
metadata locks acquired by current statement and start to wait
until conflicting lock go away. To avoid deadlocks we use simple
empiric which aborts waiting with ER_LOCK_DEADLOCK error if it
turns out that somebody is waiting for metadata locks owned by
this transaction.

This patch also fixes bug #46273 "MySQL 5.4.4 new MDL: Bug#989
is not fully fixed in case of ALTER".

The bug was that concurrent execution of UPDATE or MULTI-UPDATE
statement as a part of multi-statement transaction that already
has used table being updated and ALTER TABLE statement might have
resulted of loss of isolation between this transaction and ALTER
TABLE statement, which manifested itself as changes performed by
ALTER TABLE becoming visible in transaction and wrong binary log
order as a consequence.

This problem occurred when UPDATE or MULTI-UPDATE's wait in
mysql_lock_tables() call was aborted due to metadata lock
upgrade performed by concurrent ALTER TABLE. After such abort all
metadata locks held by transaction were released but transaction
silently continued to be executed as if nothing has happened.

We solve this problem by changing our code not to release all
locks in such case. Instead we release only locks which were
acquired by current statement and then try to reacquire them
by restarting open/lock tables process. We piggyback on simple
deadlock detector implementation since this change has to be
done anyway for it.

mysql-test/include/handler.inc:
  After introduction of basic deadlock detector for metadata locks
  it became necessary to change parts of test for HANDLER statements
  which covered some of scenarios in which ER_LOCK_DEADLOCK error
  was detected in absence of real deadlock (with new deadlock detector
  this no longer happens).
  Also adjusted test to the fact that HANDLER READ for the table no
  longer will be blocked by ALTER TABLE for the same table which awaits
  for metadata lock upgrade (this is due to removal of mysql_lock_abort()
  from wait_while_table_is_used()).
mysql-test/r/handler_innodb.result:
  After introduction of basic deadlock detector for metadata locks
  it became necessary to change parts of test for HANDLER statements
  which covered some of scenarios in which ER_LOCK_DEADLOCK error
  was detected in absence of real deadlock (with new deadlock detector
  this no longer happens).
  Also adjusted test to the fact that HANDLER READ for the table no
  longer will be blocked by ALTER TABLE for the same table which awaits
  for metadata lock upgrade (this is due to removal of mysql_lock_abort()
  from wait_while_table_is_used()).
mysql-test/r/handler_myisam.result:
  After introduction of basic deadlock detector for metadata locks
  it became necessary to change parts of test for HANDLER statements
  which covered some of scenarios in which ER_LOCK_DEADLOCK error
  was detected in absence of real deadlock (with new deadlock detector
  this no longer happens).
  Also adjusted test to the fact that HANDLER READ for the table no
  longer will be blocked by ALTER TABLE for the same table which awaits
  for metadata lock upgrade (this is due to removal of mysql_lock_abort()
  from wait_while_table_is_used()).
mysql-test/r/mdl_sync.result:
  Added test coverage for basic deadlock detection in metadata
  locking subsystem and for bug #46273 "MySQL 5.4.4 new MDL:
  Bug#989 is not fully fixed in case of ALTER".
mysql-test/r/sp-lock.result:
  Adjusted test coverage for metadata locking for stored routines
  since after introduction of basic deadlock detector for metadata
  locks number of scenarios in which ER_LOCK_DEADLOCK error in
  absence of deadlock has decreased.
mysql-test/t/mdl_sync.test:
  Added test coverage for basic deadlock detection in metadata
  locking subsystem and for bug #46273 "MySQL 5.4.4 new MDL:
  Bug#989 is not fully fixed in case of ALTER".
mysql-test/t/sp-lock.test:
  Adjusted test coverage for metadata locking for stored routines
  since after introduction of basic deadlock detector for metadata
  locks number of scenarios in which ER_LOCK_DEADLOCK error in
  absence of deadlock has decreased.
sql/log_event_old.cc:
  close_tables_for_reopen() now takes one more argument which
  specifies at which point it should stop releasing metadata
  locks acquired by this connection.
sql/mdl.cc:
  Changed metadata locking subsystem to support basic deadlock detection
  with a help of the following simple empiric -- we assume that there is
  a deadlock if there is a connection which has to wait for a metadata
  lock which is currently acquired by some connection which is itself
  waiting to be able to acquire some shared metadata lock.
  
  To implement this change:
  - Added MDL_context::can_wait_lead_to_deadlock()/_impl() methods
    which allow to find out if there is someone waiting for metadata
    lock which is held by the connection and therefore deadlocks are
    possible if this connection is going to wait for some metadata lock.
    To do this added version of MDL_ticket::has_pending_conflicting_lock()
    method which assumes that its caller already owns LOCK_mdl mutex.
  - Changed MDL_context::wait_for_locks() to use one of the above methods
    to check if somebody is waiting for metadata lock owned by this
    context (and therefore deadlock is possible) and emit ER_LOCK_DEADLOCK
    error in this case. Also now we mark context of connections waiting
    inside of this method by setting MDL_context::m_is_waiting_in_mdl
    member. Thanks to this such connection could be waken up if some
    other connection starts waiting for one of its metadata locks and
    so a deadlock can occur.
  - Adjusted notify_shared_lock() to wake up connections which wait inside
    MDL_context::wait_for_locks() while holding shared metadata lock.
  - Changed MDL_ticket::upgrade_shared_lock_to_exclusive() to add
    temporary ticket for exclusive lock to MDL_lock::waiting queue, so
    request for metadata lock upgrade can be properly detected by our
    empiric.
    Also now this method invokes a callback which forces transactions
    holding shared metadata lock on the table to call MDL_context::
    can_wait_lead_to_deadlock() method even if they don't need any new
    metadata locks. Thanks to this such transactions can detect deadlocks/
    livelocks between MDL and table-level locks.
  
  Also reduced timeouts between calls to notify_shared_lock()
  in MDL_ticket::upgrade_shared_lock_to_exclusive() and
  MDL_context::acquire_exclusive_locks(). This was necessary
  to get rid of call to mysql_lock_abort() in wait_while_table_is_used().
  (Now we instead rely on notify_shared_lock() timely calling
  mysql_lock_abort_for_thread() for the table on which lock
  is being upgraded/acquired).
sql/mdl.h:
  - Added a version of MDL_ticket::has_pending_conflicting_lock() method
    to be used in situations when caller already has acquired LOCK_mdl
    mutex.
  - Added MDL_context::can_wait_lead_to_deadlock()/_impl() methods
    which allow to find out if there is someone waiting for metadata lock
    which is held by this connection and thus deadlocks are possible if
    this connections will start waiting for some metadata lock.
  - Added MDL_context::m_is_waiting_in_mdl member to mark connections
    waiting in MDL_context::wait_for_locks() method of metadata locking
    subsystem. Added getter method for this private member to make it
    accessible in notify_shared_lock() auxiliary so we can wake-up such
    connections if they hold shared metadata locks.
  - Finally, added mysql_abort_transactions_with_shared_lock() callback
    to be able force transactions which don't need any new metadata
    locks still call MDL_context::can_wait_lead_to_deadlock() and detect
    some of deadlocks between metadata locks and table-level locks.
sql/mysql_priv.h:
  close_tables_for_reopen() now takes one more argument which
  specifies at which point it should stop releasing metadata
  locks acquired by this connection.
sql/sql_base.cc:
  Changed approach to metadata locking for multi-statement transactions.
  We no longer fail ER_LOCK_DEADLOCK error immediately when we encounter
  conflicting metadata lock. Instead we release all metadata locks
  acquired by current statement and start to wait until conflicting
  locks to go away by calling MDL_context::wait_for_locks() method.
  To avoid deadlocks the latter implements simple empiric which aborts
  waiting with ER_LOCK_DEADLOCK error if it turns out that somebody
  is waiting for metadata locks owned by this transaction.
  
  To implement the change described above:
  - Introduced Open_table_context::m_start_of_statement_svp member to
    store state of metadata locks at the start of the statement.
  - Changed Open_table_context::request_backoff_action() not to
    fail with ER_LOCK_DEADLOCK immediately if back-off is requested
    due to conflicting metadata lock.
  - Added new argument for close_tables_for_reopen() procedure which
    allows to specify subset of metadata locks to be released.
  - Changed open_tables() not to release all metadata locks acquired
    by current transaction when metadata lock conflict is discovered.
    Instead we release only locks acquired by current statement.
  - Changed open_ltable() and open_and_lock_tables_derived() not to emit
    ER_LOCK_DEADLOCK error when mysql_lock_tables() is aborted in
    multi-statement transaction when somebody tries to acquire exclusive
    metadata lock on the table. Instead we release metadata locks acquired
    by current statement and try to wait until they can be re-acquired.
  - Adjusted tdc_wait_for_old_versions() to check if there is someone
    waiting for one of metadata locks held by this connection and run
    deadlock detection in order to avoid deadlocks in some
    situations.
  - Added mysql_abort_transactions_with_shared_lock() callback which
    allows to force transactions holding shared metadata lock on the
    table to call MDL_context::can_wait_lead_to_deadlock() even if they
    don't need any new metadata locks so they can detect potential
    deadlocks between metadata locking subsystem and table-level locks.
  - Adjusted wait_while_table_is_used() not to set TABLE::version to
    0 as it is now done only when necessary by the above-mentioned
    callback. Also removed unnecessary call to mysql_lock_abort().
    Instead we rely on code performing metadata lock upgrade aborting
    waits on the table-level lock for this table by calling
    mysql_lock_abort_for_thread() (invoked by
    mysql_notify_thread_having_shared_lock()). In future this should
    allow to reduce number of scenarios in which we produce
    ER_LOCK_DEADLOCK error even though no real deadlock exists.
sql/sql_class.h:
  Introduced Open_table_context::m_start_of_statement_svp member to
  store state of metadata locks at the start of the statement.
  Replaced Open_table_context::m_can_deadlock member with m_has_locks
  member to reflect the fact that we no longer unconditionally emit
  ER_LOCK_DEADLOCK error for transaction having some metadata locks
  when conflicting metadata lock is discovered.
sql/sql_insert.cc:
  close_tables_for_reopen() now takes one more argument which
  specifies at which point it should stop releasing metadata
  locks acquired by this connection.
sql/sql_plist.h:
  Made I_P_List_iterator<T, B> usable with const lists.
sql/sql_show.cc:
  close_tables_for_reopen() now takes one more argument which
  specifies at which point it should stop releasing metadata
  locks acquired by this connection.
sql/sql_update.cc:
  Changed UPDATE and MULTI-UPDATE code not to release all metadata
  locks when calls to mysql_lock_tables() are aborted. Instead we
  release only locks which are acquired by this statement and then
  try to reacquire them by calling open_tables(). This solves
  bug #46273 "MySQL 5.4.4 new MDL: Bug#989 is not fully fixed in
  case of ALTER".

mysql-test/include/handler.inc

@@ -732,7 +732,7 @@ connection default;
 --disable_warnings
 drop table if exists t1;
 --enable_warnings
-create table t1 (a int);
+create table t1 (a int, key a (a));
 insert into t1 values (1);
 handler t1 open;
 connection con1;
@@ -743,7 +743,6 @@ let $wait_condition=
   where state = "Waiting for table" and info = "alter table t1 engine=memory";
 --source include/wait_condition.inc
 connection default;
---error ER_ILLEGAL_HA
 handler t1 read a next;
 handler t1 close;
 connection con1;
@@ -983,11 +982,12 @@ lock table t2 read;
 --echo # --> connection con2 
 connection con2;
 --echo # Sending:
---send drop table t2
+send rename table t2 to t3, t1 to t2, t3 to t1;
 --echo # --> connection con1 
 connection con1;
---echo # Waiting for 'drop table t2' to get blocked...
-let $wait_condition=select count(*)=1 from information_schema.processlist where state='Waiting for table' and info='drop table t2';
+--echo # Waiting for 'rename table ...' to get blocked...
+let $wait_condition=select count(*)=1 from information_schema.processlist
+where state='Waiting for table' and info='rename table t2 to t3, t1 to t2, t3 to t1';
 --source include/wait_condition.inc
 --echo # --> connection default
 connection default;
@@ -997,25 +997,26 @@ handler t2 open;
 select * from t2;
 handler t1 open;
 commit;
---error ER_LOCK_DEADLOCK
-handler t2 open;
 handler t1 close;
 --echo # --> connection con1
 connection con1;
 unlock tables;
 --echo # --> connection con2
 connection con2;
---echo # Reaping 'drop table t2'...
+--echo # Reaping 'rename table ...'...
 --reap
 --echo # --> connection default
 connection default;
 handler t1 open;
 handler t1 read a prev;
 handler t1 close;
+drop table t2;
 --echo #
---echo # Likewise, this doesn't require a multi-statement transaction.
---echo # ER_LOCK_DEADLOCK is also produced when we have an open
---echo # HANDLER and try to acquire locks for a single statement.
+--echo # Originally there was a deadlock error in this test.
+--echo # With implementation of deadlock detector
+--echo # we no longer deadlock, but block and wait on a lock.
+--echo # The HANDLER is auto-closed as soon as the connection
+--echo # sees a pending conflicting lock against it.
 --echo #
 create table t2 (a int, key a (a));
 handler t1 open;
@@ -1033,10 +1034,12 @@ let $wait_condition=select count(*)=1 from information_schema.processlist where
 --source include/wait_condition.inc
 --echo # --> connection default
 connection default;
---error ER_LOCK_DEADLOCK
-select * from t2;
+--echo # Sending 'select * from t2'
+send select * from t2;
 --echo # --> connection con1
 connection con1;
+--echo # Waiting for 'select * from t2' to get blocked...
+let $wait_condition=select count(*)=1 from information_schema.processlist where state='Waiting for table' and info='select * from t2';
 unlock tables;
 --echo # --> connection con2
 connection con2;
@@ -1044,6 +1047,9 @@ connection con2;
 --reap
 --echo # --> connection default
 connection default;
+--echo # Reaping 'select * from t2'
+--error ER_NO_SUCH_TABLE
+reap;
 handler t1 close;
 
 --echo #

mysql-test/r/handler_innodb.result

@@ -745,12 +745,13 @@ drop table t1;
 handler t1 read a next;
 ERROR 42S02: Unknown table 't1' in HANDLER
 drop table if exists t1;
-create table t1 (a int);
+create table t1 (a int, key a (a));
 insert into t1 values (1);
 handler t1 open;
 alter table t1 engine=memory;
 handler t1 read a next;
-ERROR HY000: Table storage engine for 't1' doesn't have this option
+a
+1
 handler t1 close;
 drop table t1;
 USE information_schema;
@@ -1002,9 +1003,9 @@ a
 lock table t2 read;
 # --> connection con2 
 # Sending:
-drop table t2;
+rename table t2 to t3, t1 to t2, t3 to t1;
 # --> connection con1 
-# Waiting for 'drop table t2' to get blocked...
+# Waiting for 'rename table ...' to get blocked...
 # --> connection default
 handler t2 open;
 ERROR 40001: Deadlock found when trying to get lock; try restarting transaction
@@ -1012,23 +1013,24 @@ select * from t2;
 ERROR 40001: Deadlock found when trying to get lock; try restarting transaction
 handler t1 open;
 commit;
-handler t2 open;
-ERROR 40001: Deadlock found when trying to get lock; try restarting transaction
 handler t1 close;
 # --> connection con1
 unlock tables;
 # --> connection con2
-# Reaping 'drop table t2'...
+# Reaping 'rename table ...'...
 # --> connection default
 handler t1 open;
 handler t1 read a prev;
 a
 5
 handler t1 close;
+drop table t2;
 #
-# Likewise, this doesn't require a multi-statement transaction.
-# ER_LOCK_DEADLOCK is also produced when we have an open
-# HANDLER and try to acquire locks for a single statement.
+# Originally there was a deadlock error in this test.
+# With implementation of deadlock detector
+# we no longer deadlock, but block and wait on a lock.
+# The HANDLER is auto-closed as soon as the connection
+# sees a pending conflicting lock against it.
 #
 create table t2 (a int, key a (a));
 handler t1 open;
@@ -1040,13 +1042,16 @@ drop table t2;
 # --> connection con1
 # Waiting for 'drop table t2' to get blocked...
 # --> connection default
+# Sending 'select * from t2'
 select * from t2;
-ERROR 40001: Deadlock found when trying to get lock; try restarting transaction
 # --> connection con1
+# Waiting for 'select * from t2' to get blocked...
 unlock tables;
 # --> connection con2
 # Reaping 'drop table t2'...
 # --> connection default
+# Reaping 'select * from t2'
+ERROR 42S02: Table 'test.t2' doesn't exist
 handler t1 close;
 #
 # ROLLBACK TO SAVEPOINT releases transactional locks,

mysql-test/r/handler_myisam.result

@@ -743,12 +743,13 @@ drop table t1;
 handler t1 read a next;
 ERROR 42S02: Unknown table 't1' in HANDLER
 drop table if exists t1;
-create table t1 (a int);
+create table t1 (a int, key a (a));
 insert into t1 values (1);
 handler t1 open;
 alter table t1 engine=memory;
 handler t1 read a next;
-ERROR HY000: Table storage engine for 't1' doesn't have this option
+a
+1
 handler t1 close;
 drop table t1;
 USE information_schema;
@@ -999,9 +1000,9 @@ a
 lock table t2 read;
 # --> connection con2 
 # Sending:
-drop table t2;
+rename table t2 to t3, t1 to t2, t3 to t1;
 # --> connection con1 
-# Waiting for 'drop table t2' to get blocked...
+# Waiting for 'rename table ...' to get blocked...
 # --> connection default
 handler t2 open;
 ERROR 40001: Deadlock found when trying to get lock; try restarting transaction
@@ -1009,23 +1010,24 @@ select * from t2;
 ERROR 40001: Deadlock found when trying to get lock; try restarting transaction
 handler t1 open;
 commit;
-handler t2 open;
-ERROR 40001: Deadlock found when trying to get lock; try restarting transaction
 handler t1 close;
 # --> connection con1
 unlock tables;
 # --> connection con2
-# Reaping 'drop table t2'...
+# Reaping 'rename table ...'...
 # --> connection default
 handler t1 open;
 handler t1 read a prev;
 a
 5
 handler t1 close;
+drop table t2;
 #
-# Likewise, this doesn't require a multi-statement transaction.
-# ER_LOCK_DEADLOCK is also produced when we have an open
-# HANDLER and try to acquire locks for a single statement.
+# Originally there was a deadlock error in this test.
+# With implementation of deadlock detector
+# we no longer deadlock, but block and wait on a lock.
+# The HANDLER is auto-closed as soon as the connection
+# sees a pending conflicting lock against it.
 #
 create table t2 (a int, key a (a));
 handler t1 open;
@@ -1037,13 +1039,16 @@ drop table t2;
 # --> connection con1
 # Waiting for 'drop table t2' to get blocked...
 # --> connection default
+# Sending 'select * from t2'
 select * from t2;
-ERROR 40001: Deadlock found when trying to get lock; try restarting transaction
 # --> connection con1
+# Waiting for 'select * from t2' to get blocked...
 unlock tables;
 # --> connection con2
 # Reaping 'drop table t2'...
 # --> connection default
+# Reaping 'select * from t2'
+ERROR 42S02: Table 'test.t2' doesn't exist
 handler t1 close;
 #
 # ROLLBACK TO SAVEPOINT releases transactional locks,

mysql-test/r/mdl_sync.result

@@ -20,6 +20,220 @@ ERROR 42S02: Unknown table 't1'
 drop table t3;
 SET DEBUG_SYNC= 'RESET';
 #
+# Test coverage for basic deadlock detection in metadata
+# locking subsystem.
+#
+drop tables if exists t1, t2, t3, t4;
+create table t1 (i int);
+create table t2 (j int);
+create table t3 (k int);
+create table t4 (k int);
+#
+# Test for the case in which no deadlock occurs.
+#
+#
+# Switching to connection 'deadlock_con1'.
+begin;
+insert into t1 values (1);
+#
+# Switching to connection 'deadlock_con2'.
+begin;
+insert into t2 values (1);
+#
+# Switching to connection 'default'.
+# Send:
+rename table t2 to t0, t3 to t2, t0 to t3;;
+#
+# Switching to connection 'deadlock_con1'.
+# Wait until the above RENAME TABLE is blocked because it has to wait
+# for 'deadlock_con2' which holds shared metadata lock on 't2'.
+# The below statement should wait for exclusive metadata lock
+# on 't2' to go away and should not produce ER_LOCK_DEADLOCK
+# as no deadlock is possible in this situation.
+# Send:
+select * from t2;;
+#
+# Switching to connection 'deadlock_con2'.
+# Wait until the above SELECT * FROM t2 is starts waiting
+# for an exclusive metadata lock to go away.
+#
+# Unblock RENAME TABLE by releasing shared metadata lock on t2.
+commit;
+#
+# Switching to connection 'default'.
+# Reap RENAME TABLE.
+#
+# Switching to connection 'deadlock_con1'.
+# Reap SELECT.
+k
+#
+# Switching to connection 'default'.
+#
+# Let us check that in the process of waiting for conflicting lock
+# on table 't2' to go away transaction in connection 'deadlock_con1'
+# has not released metadata lock on table 't1'.
+# Send:
+rename table t1 to t0, t3 to t1, t0 to t3;;
+#
+# Switching to connection 'deadlock_con1'.
+# Wait until the above RENAME TABLE is blocked because it has to wait
+# for 'deadlock_con1' which should still hold shared metadata lock on
+# table 't1'.
+# Commit transaction to unblock RENAME TABLE.
+commit;
+#
+# Switching to connection 'default'.
+# Reap RENAME TABLE.
+#
+# Test for case when deadlock occurs and should be detected immediately.
+#
+#
+# Switching to connection 'deadlock_con1'.
+begin;
+insert into t1 values (2);
+#
+# Switching to connection 'default'.
+# Send:
+rename table t2 to t0, t1 to t2, t0 to t1;;
+#
+# Switching to connection 'deadlock_con1'.
+# Wait until the above RENAME TABLE is blocked because it has to wait
+# for 'deadlock_con1' which holds shared metadata lock on 't1'.
+# 
+# The below statement should not wait as doing so will cause deadlock.
+# Instead it should fail and emit ER_LOCK_DEADLOCK statement.
+select * from t2;
+ERROR 40001: Deadlock found when trying to get lock; try restarting transaction
+#
+# Let us check that failure of the above statement has not released
+# metadata lock on table 't1', i.e. that RENAME TABLE is still blocked.
+# Commit transaction to unblock RENAME TABLE.
+commit;
+#
+# Switching to connection 'default'.
+# Reap RENAME TABLE.
+#
+# Test for the case in which deadlock also occurs but not immediately.
+#
+#
+# Switching to connection 'deadlock_con1'.
+begin;
+insert into t1 values (1);
+#
+# Switching to connection 'deadlock_con2'.
+begin;
+insert into t3 values (1);
+#
+# Switching to connection 'default'.
+# Send:
+rename table t2 to t0, t3 to t2, t0 to t3;;
+#
+# Switching to connection 'deadlock_con1'.
+# Wait until the above RENAME TABLE is blocked because it has to wait
+# for 'deadlock_con2' which holds shared metadata lock on 't3'.
+# The below SELECT statement should wait for metadata lock
+# on table 't2' and should not produce ER_LOCK_DEADLOCK
+# immediately as no deadlock is possible at the moment.
+select * from t2;;
+#
+# Switching to connection 'deadlock_con3'.
+# Wait until the above SELECT * FROM t2 is starts waiting
+# for an exclusive metadata lock to go away.
+# Send RENAME TABLE statement that will deadlock with the
+# SELECT statement and thus should abort the latter.
+rename table t1 to t0, t2 to t1, t0 to t2;;
+#
+# Switching to connection 'deadlock_con1'.
+# Since the latest RENAME TABLE entered in deadlock with SELECT
+# statement the latter should be aborted and emit ER_LOCK_DEADLOCK
+# error.
+# Reap SELECT * FROM t2.
+ERROR 40001: Deadlock found when trying to get lock; try restarting transaction
+#
+# Again let us check that failure of the SELECT statement has not
+# released metadata lock on table 't1', i.e. that the latest RENAME
+# is blocked.
+# Commit transaction to unblock this RENAME TABLE.
+commit;
+#
+# Switching to connection 'deadlock_con3'.
+# Reap RENAME TABLE t1 TO t0 ... .
+#
+# Switching to connection 'deadlock_con2'.
+# Commit transaction to unblock the first RENAME TABLE.
+commit;
+#
+# Switching to connection 'default'.
+# Reap RENAME TABLE t2 TO t0 ... .
+drop tables t1, t2, t3, t4;
+#
+# Now, test case which shows that deadlock detection empiric
+# also takes into account requests for metadata lock upgrade.
+#
+create table t1 (i int);
+#
+# Switching to connection 'deadlock_con1'.
+begin;
+insert into t1 values (1);
+#
+# Switching to connection 'default'.
+# Send:
+alter table t1 add column j int, rename to t2;;
+#
+# Switching to connection 'deadlock_con1'.
+# Wait until the above ALTER TABLE ... RENAME acquires exclusive
+# metadata lock on 't2' and starts waiting for connection
+# 'deadlock_con1' which holds shared lock on 't1'.
+# The below statement should not wait as it will cause deadlock.
+# An appropriate error should be reported instead.
+select * from t2;
+ERROR 40001: Deadlock found when trying to get lock; try restarting transaction
+# Again let us check that failure of the above statement has not
+# released all metadata locks in connection 'deadlock_con1' and
+# so ALTER TABLE ... RENAME is still blocked.
+# Commit transaction to unblock ALTER TABLE ... RENAME.
+commit;
+#
+# Switching to connection 'default'.
+# Reap ALTER TABLE ... RENAME.
+drop table t2;
+#
+# Finally, test case in which deadlock (or potentially livelock) occurs
+# between metadata locking subsystem and table definition cache/table
+# locks, but which should still be detected by our empiric.
+#
+create table t1 (i int);
+#
+# Switching to connection 'deadlock_con1'.
+begin;
+insert into t1 values (1);
+#
+# Switching to connection 'default'.
+lock tables t1 write;
+#
+# Switching to connection 'deadlock_con1'.
+# Send:
+insert into t1 values (2);;
+#
+# Switching to connection 'default'.
+# Wait until INSERT in connection 'deadlock_con1' is blocked on
+# table-level lock.
+# Send:
+alter table t1 add column j int;;
+#
+# Switching to connection 'deadlock_con1'.
+# The above ALTER TABLE statement should cause INSERT statement in
+# this connection to be aborted and emit ER_LOCK_DEADLOCK error.
+# Reap INSERT
+ERROR 40001: Deadlock found when trying to get lock; try restarting transaction
+# Commit transaction to unblock ALTER TABLE.
+commit;
+#
+# Switching to connection 'default'.
+# Reap ALTER TABLE.
+unlock tables;
+drop table t1;
+#
 # Test for bug #46748 "Assertion in MDL_context::wait_for_locks()
 # on INSERT + CREATE TRIGGER".
 #
@@ -234,6 +448,43 @@ drop table t2;
 # Clean-up.
 drop table t1;
 #
+# Test for bug #46273 "MySQL 5.4.4 new MDL: Bug#989 is not fully fixed
+#                      in case of ALTER".
+#
+drop table if exists t1;
+set debug_sync= 'RESET';
+create table t1 (c1 int primary key, c2 int, c3 int);
+insert into t1 values (1,1,0),(2,2,0),(3,3,0),(4,4,0),(5,5,0);
+begin;
+update t1 set c3=c3+1 where c2=3;
+#
+# Switching to connection 'con46273'.
+set debug_sync='after_lock_tables_takes_lock SIGNAL alter_table_locked WAIT_FOR alter_go';
+alter table t1 add column e int, rename to t2;;
+#
+# Switching to connection 'default'.
+set debug_sync='now WAIT_FOR alter_table_locked';
+set debug_sync='wait_for_lock SIGNAL alter_go';
+# The below statement should get ER_LOCK_DEADLOCK error
+# (i.e. it should not allow ALTER to proceed, and then
+# fail due to 't1' changing its name to 't2').
+update t1 set c3=c3+1 where c2=4;
+ERROR 40001: Deadlock found when trying to get lock; try restarting transaction
+#
+# Let us check that failure of the above statement has not released
+# metadata lock on table 't1', i.e. that ALTER TABLE is still blocked.
+# Unblock ALTER TABLE by commiting transaction and thus releasing
+# metadata lock on 't1'.
+commit;
+#
+# Switching to connection 'con46273'.
+# Reap ALTER TABLE.
+#
+# Switching to connection 'default'.
+# Clean-up.
+set debug_sync= 'RESET';
+drop table t2;
+#
 # Test for bug #46673 "Deadlock between FLUSH TABLES WITH READ LOCK
 #                      and DML".
 #

mysql-test/r/sp-lock.result

@@ -125,16 +125,15 @@ drop temporary table t1;
 #
 # For that, start a transaction, use a routine. In a concurrent
 # connection, try to drop or alter the routine. It should place
-# a pending or exlusive lock and block. In a concurrnet
-# connection, try to use the routine under LOCK TABLES. 
-# That should yield ER_LOCK_DEADLOCK.
+# a pending or exclusive lock and block. In another concurrnet
+# connection, try to use the routine.
+# That should block on the pending exclusive lock.
 #
 # Establish helper connections.
 #
 # Test DROP PROCEDURE.
 #
 # --> connection default
-create table t1 (a int);
 create procedure p1() begin end;
 create function f1() returns int 
 begin
@@ -151,14 +150,17 @@ drop procedure p1;
 # --> connection con2 
 # Waitng for 'drop procedure t1' to get blocked on MDL lock...
 # Demonstrate that there is a pending exclusive lock.
-lock table t1 read;
+# Sending 'select f1()'...
 select f1();
-ERROR 40001: Deadlock found when trying to get lock; try restarting transaction
-unlock tables;
+# --> connection con3
+# Waitng for 'select f1()' to get blocked by a pending MDL lock...
 # --> connection default
 commit;
 # --> connection con1
 # Reaping 'drop procedure p1'...
+# --> connection con2
+# Reaping 'select f1()'
+ERROR 42000: PROCEDURE test.p1 does not exist
 # --> connection default
 #
 # Test CREATE PROCEDURE.
@@ -174,17 +176,22 @@ create procedure p1() begin end;
 # --> connection con2 
 # Waitng for 'create procedure t1' to get blocked on MDL lock...
 # Demonstrate that there is a pending exclusive lock.
-lock table t1 read;
+# Sending 'select f1()'...
 select f1();
-ERROR 40001: Deadlock found when trying to get lock; try restarting transaction
-unlock tables;
+# --> connection con3
+# Waitng for 'select f1()' to get blocked by a pending MDL lock...
 # --> connection default
 commit;
 # --> connection con1
 # Reaping 'create procedure p1'...
 ERROR 42000: PROCEDURE p1 already exists
+# --> connection con2
+# Reaping 'select f1()'
+f1()
+1
 # 
 # Test ALTER PROCEDURE.
+#
 begin;
 select f1();
 f1()
@@ -195,14 +202,18 @@ alter procedure p1 contains sql;
 # --> connection con2 
 # Waitng for 'alter procedure t1' to get blocked on MDL lock...
 # Demonstrate that there is a pending exclusive lock.
-lock table t1 read;
+# Sending 'select f1()'...
 select f1();
-ERROR 40001: Deadlock found when trying to get lock; try restarting transaction
-unlock tables;
+# --> connection con3
+# Waitng for 'select f1()' to get blocked by a pending MDL lock...
 # --> connection default
 commit;
 # --> connection con1
 # Reaping 'alter procedure p1'...
+# --> connection con2
+# Reaping 'select f1()'
+f1()
+1
 # --> connection default
 #
 # Test DROP FUNCTION.
@@ -217,14 +228,17 @@ drop function f1;
 # --> connection con2 
 # Waitng for 'drop function f1' to get blocked on MDL lock...
 # Demonstrate that there is a pending exclusive lock.
-lock table t1 read;
+# Sending 'select f1()'...
 select f1();
-ERROR 40001: Deadlock found when trying to get lock; try restarting transaction
-unlock tables;
+# --> connection con3
+# Waitng for 'select f1()' to get blocked by a pending MDL lock...
 # --> connection default
 commit;
 # --> connection con1
 # Reaping 'drop function f1'...
+# --> connection con2
+# Reaping 'select f1()'
+ERROR 42000: FUNCTION test.f1 does not exist
 # --> connection default
 #
 # Test CREATE FUNCTION.
@@ -240,18 +254,23 @@ create function f1() returns int return 2;
 # --> connection con2 
 # Waitng for 'create function f1' to get blocked on MDL lock...
 # Demonstrate that there is a pending exclusive lock.
-lock table t1 read;
+# Sending 'select f1()'...
 select f1();
-ERROR 40001: Deadlock found when trying to get lock; try restarting transaction
-unlock tables;
+# --> connection con3
+# Waitng for 'select f1()' to get blocked by a pending MDL lock...
 # --> connection default
 commit;
 # --> connection con1
 # Reaping 'create function f1'...
 ERROR 42000: FUNCTION f1 already exists
+# --> connection con2
+# Reaping 'select f1()'
+f1()
+1
 # --> connection default
 # 
 # Test ALTER FUNCTION.
+# 
 begin;
 select f1();
 f1()
@@ -262,14 +281,18 @@ alter function f1 contains sql;
 # --> connection con2 
 # Waitng for 'alter function f1' to get blocked on MDL lock...
 # Demonstrate that there is a pending exclusive lock.
-lock table t1 read;
+# Sending 'select f1()'...
 select f1();
-ERROR 40001: Deadlock found when trying to get lock; try restarting transaction
-unlock tables;
+# --> connection con3
+# Waitng for 'select f1()' to get blocked by a pending MDL lock...
 # --> connection default
 commit;
 # --> connection con1
 # Reaping 'alter function f1'...
+# --> connection con2
+# Reaping 'select f1()'
+f1()
+1
 # --> connection default
 drop function f1;
 drop procedure p1;
@@ -283,6 +306,7 @@ drop procedure p1;
 #
 create procedure p1() begin end;
 create procedure p2() begin end;
+create table t1 (a int);
 create procedure p3()
 begin
 call p1();
@@ -415,36 +439,11 @@ drop table t1, t2;
 # acquisition of a shared lock fails during a transaction or
 # we need to back off to flush the sp cache.
 #
-# a) A back off due to a lock conflict.
-#
-create table t1 (a int);
-create function f1() returns int return 6;
-begin;
-select f1();
-f1()
-6
-# --> connection con1
-# Sending 'drop function f1'...
-drop function f1;
-# --> connection con2
-# Waitng for 'drop function f1' to get blocked on MDL lock...
-begin;
-select * from t1;
-a
-select f1();
-ERROR 40001: Deadlock found when trying to get lock; try restarting transaction
-commit;
-# --> connection default
-commit;
-# --> connection con1
-# Reaping 'drop function f1'...
-# --> connection default
-#
-# b) A back off to flush the cache.
 # Sic: now this situation does not require a back off since we
 # flush the cache on the fly.
 #
 create function f1() returns int return 7;
+create table t1 (a int);
 begin;
 select * from t1;
 a
@@ -691,6 +690,7 @@ drop function f1;
 set @@session.max_sp_recursion_depth=default;
 # --> connection con1
 # --> connection con2
+# --> connection con3
 # --> connection default
 #
 # End of 5.5 tests

mysql-test/t/mdl_sync.test

@@ -73,6 +73,366 @@ SET DEBUG_SYNC= 'RESET';
 --enable_warnings
 
 
+--echo #
+--echo # Test coverage for basic deadlock detection in metadata
+--echo # locking subsystem.
+--echo #
+--disable_warnings
+drop tables if exists t1, t2, t3, t4;
+--enable_warnings
+
+connect(deadlock_con1,localhost,root,,);
+connect(deadlock_con2,localhost,root,,);
+connect(deadlock_con3,localhost,root,,);
+connection default;
+create table t1 (i int);
+create table t2 (j int);
+create table t3 (k int);
+create table t4 (k int);
+
+--echo #
+--echo # Test for the case in which no deadlock occurs.
+--echo #
+
+--echo #
+--echo # Switching to connection 'deadlock_con1'.
+connection deadlock_con1;
+begin;
+insert into t1 values (1);
+
+--echo #
+--echo # Switching to connection 'deadlock_con2'.
+connection deadlock_con2;
+begin;
+insert into t2 values (1);
+
+--echo #
+--echo # Switching to connection 'default'.
+connection default;
+--echo # Send:
+--send rename table t2 to t0, t3 to t2, t0 to t3;
+
+--echo #
+--echo # Switching to connection 'deadlock_con1'.
+connection deadlock_con1;
+--echo # Wait until the above RENAME TABLE is blocked because it has to wait
+--echo # for 'deadlock_con2' which holds shared metadata lock on 't2'.
+let $wait_condition=
+  select count(*) = 1 from information_schema.processlist
+  where state = "Waiting for table" and info = "rename table t2 to t0, t3 to t2, t0 to t3";
+--source include/wait_condition.inc
+--echo # The below statement should wait for exclusive metadata lock
+--echo # on 't2' to go away and should not produce ER_LOCK_DEADLOCK
+--echo # as no deadlock is possible in this situation.
+--echo # Send:
+--send select * from t2;
+
+--echo #
+--echo # Switching to connection 'deadlock_con2'.
+connection deadlock_con2;
+--echo # Wait until the above SELECT * FROM t2 is starts waiting
+--echo # for an exclusive metadata lock to go away.
+let $wait_condition=
+  select count(*) = 1 from information_schema.processlist
+  where state = "Waiting for table" and info = "select * from t2";
+--source include/wait_condition.inc
+--echo #
+--echo # Unblock RENAME TABLE by releasing shared metadata lock on t2.
+commit;
+
+--echo #
+--echo # Switching to connection 'default'.
+connection default;
+--echo # Reap RENAME TABLE.
+--reap
+
+--echo #
+--echo # Switching to connection 'deadlock_con1'.
+connection deadlock_con1;
+--echo # Reap SELECT.
+--reap
+
+--echo #
+--echo # Switching to connection 'default'.
+connection default;
+--echo #
+--echo # Let us check that in the process of waiting for conflicting lock
+--echo # on table 't2' to go away transaction in connection 'deadlock_con1'
+--echo # has not released metadata lock on table 't1'.
+--echo # Send:
+--send rename table t1 to t0, t3 to t1, t0 to t3;
+
+--echo #
+--echo # Switching to connection 'deadlock_con1'.
+connection deadlock_con1;
+--echo # Wait until the above RENAME TABLE is blocked because it has to wait
+--echo # for 'deadlock_con1' which should still hold shared metadata lock on
+--echo # table 't1'.
+let $wait_condition=
+  select count(*) = 1 from information_schema.processlist
+  where state = "Waiting for table" and info = "rename table t1 to t0, t3 to t1, t0 to t3";
+--source include/wait_condition.inc
+--echo # Commit transaction to unblock RENAME TABLE.
+commit;
+
+--echo #
+--echo # Switching to connection 'default'.
+connection default;
+--echo # Reap RENAME TABLE.
+--reap
+
+--echo #
+--echo # Test for case when deadlock occurs and should be detected immediately.
+--echo #
+
+--echo #
+--echo # Switching to connection 'deadlock_con1'.
+connection deadlock_con1;
+begin;
+insert into t1 values (2);
+
+--echo #
+--echo # Switching to connection 'default'.
+connection default;
+--echo # Send:
+--send rename table t2 to t0, t1 to t2, t0 to t1;
+
+--echo #
+--echo # Switching to connection 'deadlock_con1'.
+connection deadlock_con1;
+--echo # Wait until the above RENAME TABLE is blocked because it has to wait
+--echo # for 'deadlock_con1' which holds shared metadata lock on 't1'.
+let $wait_condition=
+  select count(*) = 1 from information_schema.processlist
+  where state = "Waiting for table" and info = "rename table t2 to t0, t1 to t2, t0 to t1";
+--source include/wait_condition.inc
+--echo # 
+--echo # The below statement should not wait as doing so will cause deadlock.
+--echo # Instead it should fail and emit ER_LOCK_DEADLOCK statement.
+--error ER_LOCK_DEADLOCK
+select * from t2;
+
+--echo #
+--echo # Let us check that failure of the above statement has not released
+--echo # metadata lock on table 't1', i.e. that RENAME TABLE is still blocked.
+let $wait_condition=
+  select count(*) = 1 from information_schema.processlist
+  where state = "Waiting for table" and info = "rename table t2 to t0, t1 to t2, t0 to t1";
+--source include/wait_condition.inc
+--echo # Commit transaction to unblock RENAME TABLE.
+commit;
+
+--echo #
+--echo # Switching to connection 'default'.
+connection default;
+--echo # Reap RENAME TABLE.
+--reap
+
+--echo #
+--echo # Test for the case in which deadlock also occurs but not immediately.
+--echo #
+
+--echo #
+--echo # Switching to connection 'deadlock_con1'.
+connection deadlock_con1;
+begin;
+insert into t1 values (1);
+
+--echo #
+--echo # Switching to connection 'deadlock_con2'.
+connection deadlock_con2;
+begin;
+insert into t3 values (1);
+
+--echo #
+--echo # Switching to connection 'default'.
+connection default;
+--echo # Send:
+--send rename table t2 to t0, t3 to t2, t0 to t3;
+
+--echo #
+--echo # Switching to connection 'deadlock_con1'.
+connection deadlock_con1;
+--echo # Wait until the above RENAME TABLE is blocked because it has to wait
+--echo # for 'deadlock_con2' which holds shared metadata lock on 't3'.
+let $wait_condition=
+  select count(*) = 1 from information_schema.processlist
+  where state = "Waiting for table" and info = "rename table t2 to t0, t3 to t2, t0 to t3";
+--source include/wait_condition.inc
+--echo # The below SELECT statement should wait for metadata lock
+--echo # on table 't2' and should not produce ER_LOCK_DEADLOCK
+--echo # immediately as no deadlock is possible at the moment.
+--send select * from t2;
+
+--echo #
+--echo # Switching to connection 'deadlock_con3'.
+connection deadlock_con3;
+--echo # Wait until the above SELECT * FROM t2 is starts waiting
+--echo # for an exclusive metadata lock to go away.
+let $wait_condition=
+  select count(*) = 1 from information_schema.processlist
+  where state = "Waiting for table" and info = "select * from t2";
+--source include/wait_condition.inc
+
+--echo # Send RENAME TABLE statement that will deadlock with the
+--echo # SELECT statement and thus should abort the latter.
+--send rename table t1 to t0, t2 to t1, t0 to t2;
+
+--echo #
+--echo # Switching to connection 'deadlock_con1'.
+connection deadlock_con1;
+--echo # Since the latest RENAME TABLE entered in deadlock with SELECT
+--echo # statement the latter should be aborted and emit ER_LOCK_DEADLOCK
+--echo # error.
+--echo # Reap SELECT * FROM t2.
+--error ER_LOCK_DEADLOCK
+--reap
+
+--echo #
+--echo # Again let us check that failure of the SELECT statement has not
+--echo # released metadata lock on table 't1', i.e. that the latest RENAME
+--echo # is blocked.
+let $wait_condition=
+  select count(*) = 1 from information_schema.processlist
+  where state = "Waiting for table" and info = "rename table t1 to t0, t2 to t1, t0 to t2";
+--source include/wait_condition.inc
+--echo # Commit transaction to unblock this RENAME TABLE.
+commit;
+
+--echo #
+--echo # Switching to connection 'deadlock_con3'.
+connection deadlock_con3;
+--echo # Reap RENAME TABLE t1 TO t0 ... .
+--reap;
+
+--echo #
+--echo # Switching to connection 'deadlock_con2'.
+connection deadlock_con2;
+--echo # Commit transaction to unblock the first RENAME TABLE.
+commit;
+
+--echo #
+--echo # Switching to connection 'default'.
+connection default;
+--echo # Reap RENAME TABLE t2 TO t0 ... .
+--reap
+
+drop tables t1, t2, t3, t4;
+
+--echo #
+--echo # Now, test case which shows that deadlock detection empiric
+--echo # also takes into account requests for metadata lock upgrade.
+--echo #
+create table t1 (i int);
+
+--echo #
+--echo # Switching to connection 'deadlock_con1'.
+connection deadlock_con1;
+begin;
+insert into t1 values (1);
+
+--echo #
+--echo # Switching to connection 'default'.
+connection default;
+--echo # Send:
+--send alter table t1 add column j int, rename to t2;
+
+--echo #
+--echo # Switching to connection 'deadlock_con1'.
+connection deadlock_con1;
+--echo # Wait until the above ALTER TABLE ... RENAME acquires exclusive
+--echo # metadata lock on 't2' and starts waiting for connection
+--echo # 'deadlock_con1' which holds shared lock on 't1'.
+let $wait_condition=
+  select count(*) = 1 from information_schema.processlist
+  where state = "Waiting for table" and info = "alter table t1 add column j int, rename to t2";
+--source include/wait_condition.inc
+
+--echo # The below statement should not wait as it will cause deadlock.
+--echo # An appropriate error should be reported instead.
+--error ER_LOCK_DEADLOCK
+select * from t2;
+
+--echo # Again let us check that failure of the above statement has not
+--echo # released all metadata locks in connection 'deadlock_con1' and
+--echo # so ALTER TABLE ... RENAME is still blocked.
+let $wait_condition=
+  select count(*) = 1 from information_schema.processlist
+  where state = "Waiting for table" and info = "alter table t1 add column j int, rename to t2";
+--source include/wait_condition.inc
+
+--echo # Commit transaction to unblock ALTER TABLE ... RENAME.
+commit;
+
+--echo #
+--echo # Switching to connection 'default'.
+connection default;
+--echo # Reap ALTER TABLE ... RENAME.
+--reap
+
+drop table t2;
+
+--echo #
+--echo # Finally, test case in which deadlock (or potentially livelock) occurs
+--echo # between metadata locking subsystem and table definition cache/table
+--echo # locks, but which should still be detected by our empiric.
+--echo #
+create table t1 (i int);
+
+--echo #
+--echo # Switching to connection 'deadlock_con1'.
+connection deadlock_con1;
+begin;
+insert into t1 values (1);
+
+--echo #
+--echo # Switching to connection 'default'.
+connection default;
+lock tables t1 write;
+
+--echo #
+--echo # Switching to connection 'deadlock_con1'.
+connection deadlock_con1;
+--echo # Send:
+--send insert into t1 values (2);
+
+--echo #
+--echo # Switching to connection 'default'.
+connection default;
+--echo # Wait until INSERT in connection 'deadlock_con1' is blocked on
+--echo # table-level lock.
+let $wait_condition=
+  select count(*) = 1 from information_schema.processlist
+  where state = "Table lock" and info = "insert into t1 values (2)";
+--source include/wait_condition.inc
+
+--echo # Send:
+--send alter table t1 add column j int;
+
+--echo #
+--echo # Switching to connection 'deadlock_con1'.
+connection deadlock_con1;
+--echo # The above ALTER TABLE statement should cause INSERT statement in
+--echo # this connection to be aborted and emit ER_LOCK_DEADLOCK error.
+--echo # Reap INSERT
+--error ER_LOCK_DEADLOCK
+--reap
+--echo # Commit transaction to unblock ALTER TABLE.
+commit;
+
+--echo #
+--echo # Switching to connection 'default'.
+connection default;
+--echo # Reap ALTER TABLE.
+--reap
+unlock tables;
+
+drop table t1;
+disconnect deadlock_con1;
+disconnect deadlock_con2;
+disconnect deadlock_con3;
+
+
 --echo #
 --echo # Test for bug #46748 "Assertion in MDL_context::wait_for_locks()
 --echo # on INSERT + CREATE TRIGGER".
@@ -439,6 +799,66 @@ disconnect con46044_2;
 drop table t1;
 
 
+--echo #
+--echo # Test for bug #46273 "MySQL 5.4.4 new MDL: Bug#989 is not fully fixed
+--echo #                      in case of ALTER".
+--echo #
+--disable_warnings
+drop table if exists t1;
+--enable_warnings
+set debug_sync= 'RESET';
+connect (con46273,localhost,root,,test,,);
+connection default;
+create table t1 (c1 int primary key, c2 int, c3 int);
+insert into t1 values (1,1,0),(2,2,0),(3,3,0),(4,4,0),(5,5,0);
+
+begin;
+update t1 set c3=c3+1 where c2=3;
+
+--echo #
+--echo # Switching to connection 'con46273'.
+connection con46273;
+set debug_sync='after_lock_tables_takes_lock SIGNAL alter_table_locked WAIT_FOR alter_go';
+--send alter table t1 add column e int, rename to t2;
+
+--echo #
+--echo # Switching to connection 'default'.
+connection default;
+set debug_sync='now WAIT_FOR alter_table_locked';
+set debug_sync='wait_for_lock SIGNAL alter_go';
+--echo # The below statement should get ER_LOCK_DEADLOCK error
+--echo # (i.e. it should not allow ALTER to proceed, and then
+--echo # fail due to 't1' changing its name to 't2').
+--error ER_LOCK_DEADLOCK
+update t1 set c3=c3+1 where c2=4;
+
+--echo #
+--echo # Let us check that failure of the above statement has not released
+--echo # metadata lock on table 't1', i.e. that ALTER TABLE is still blocked.
+let $wait_condition=
+  select count(*) = 1 from information_schema.processlist
+  where state = "Waiting for table" and info = "alter table t1 add column e int, rename to t2";
+--source include/wait_condition.inc
+
+--echo # Unblock ALTER TABLE by commiting transaction and thus releasing
+--echo # metadata lock on 't1'.
+commit;
+
+--echo #
+--echo # Switching to connection 'con46273'.
+connection con46273;
+--echo # Reap ALTER TABLE.
+--reap
+
+--echo #
+--echo # Switching to connection 'default'.
+connection default;
+disconnect con46273;
+--echo # Clean-up.
+set debug_sync= 'RESET';
+drop table t2;
+
+
 --echo #
 --echo # Test for bug #46673 "Deadlock between FLUSH TABLES WITH READ LOCK
 --echo #                      and DML".

mysql-test/t/sp-lock.test

@@ -153,20 +153,20 @@ drop temporary table t1;
 --echo #
 --echo # For that, start a transaction, use a routine. In a concurrent
 --echo # connection, try to drop or alter the routine. It should place
---echo # a pending or exlusive lock and block. In a concurrnet
---echo # connection, try to use the routine under LOCK TABLES. 
---echo # That should yield ER_LOCK_DEADLOCK.
+--echo # a pending or exclusive lock and block. In another concurrnet
+--echo # connection, try to use the routine.
+--echo # That should block on the pending exclusive lock.
 --echo #
 --echo # Establish helper connections.
 connect(con1, localhost, root,,);
 connect(con2, localhost, root,,);
+connect(con3, localhost, root,,);
 
 --echo #
 --echo # Test DROP PROCEDURE.
 --echo #
 --echo # --> connection default
 connection default;
-create table t1 (a int);
 create procedure p1() begin end;
 delimiter |;
 create function f1() returns int 
@@ -180,7 +180,7 @@ select f1();
 --echo # --> connection con1
 connection con1;
 --echo # Sending 'drop procedure p1'...
---send drop procedure p1
+send drop procedure p1;
 --echo # --> connection con2 
 connection con2;
 --echo # Waitng for 'drop procedure t1' to get blocked on MDL lock...
@@ -188,17 +188,25 @@ let $wait_condition=select count(*)=1 from information_schema.processlist
 where state='Waiting for table' and info='drop procedure p1';
 --source include/wait_condition.inc
 --echo # Demonstrate that there is a pending exclusive lock.
-lock table t1 read;
---error ER_LOCK_DEADLOCK
-select f1();
-unlock tables;
+--echo # Sending 'select f1()'...
+send select f1();
+--echo # --> connection con3
+connection con3;
+--echo # Waitng for 'select f1()' to get blocked by a pending MDL lock...
+let $wait_condition=select count(*)=1 from information_schema.processlist
+where state='Waiting for table' and info='select f1()';
 --echo # --> connection default
 connection default;
 commit;
 --echo # --> connection con1
 connection con1;
 --echo # Reaping 'drop procedure p1'...
---reap
+reap;
+--echo # --> connection con2
+connection con2;
+--echo # Reaping 'select f1()'
+--error ER_SP_DOES_NOT_EXIST
+reap;
 --echo # --> connection default
 connection default;
 
@@ -211,7 +219,7 @@ select f1();
 --echo # --> connection con1
 connection con1;
 --echo # Sending 'create procedure p1'...
---send create procedure p1() begin end
+send create procedure p1() begin end;
 --echo # --> connection con2 
 connection con2;
 --echo # Waitng for 'create procedure t1' to get blocked on MDL lock...
@@ -219,10 +227,13 @@ let $wait_condition=select count(*)=1 from information_schema.processlist
 where state='Waiting for table' and info='create procedure p1() begin end';
 --source include/wait_condition.inc
 --echo # Demonstrate that there is a pending exclusive lock.
-lock table t1 read;
---error ER_LOCK_DEADLOCK
-select f1();
-unlock tables;
+--echo # Sending 'select f1()'...
+send select f1();
+--echo # --> connection con3
+connection con3;
+--echo # Waitng for 'select f1()' to get blocked by a pending MDL lock...
+let $wait_condition=select count(*)=1 from information_schema.processlist
+where state='Waiting for table' and info='select f1()';
 --echo # --> connection default
 connection default;
 commit;
@@ -230,16 +241,22 @@ commit;
 connection con1;
 --echo # Reaping 'create procedure p1'...
 --error ER_SP_ALREADY_EXISTS
---reap
+reap;
+--echo # --> connection con2
+connection con2;
+--echo # Reaping 'select f1()'
+reap;
 connection default;
+
 --echo # 
 --echo # Test ALTER PROCEDURE.
+--echo #
 begin;
 select f1();
 --echo # --> connection con1
 connection con1;
 --echo # Sending 'alter procedure p1'...
---send alter procedure p1 contains sql
+send alter procedure p1 contains sql;
 --echo # --> connection con2 
 connection con2;
 --echo # Waitng for 'alter procedure t1' to get blocked on MDL lock...
@@ -247,17 +264,24 @@ let $wait_condition=select count(*)=1 from information_schema.processlist
 where state='Waiting for table' and info='alter procedure p1 contains sql';
 --source include/wait_condition.inc
 --echo # Demonstrate that there is a pending exclusive lock.
-lock table t1 read;
---error ER_LOCK_DEADLOCK
-select f1();
-unlock tables;
+--echo # Sending 'select f1()'...
+send select f1();
+--echo # --> connection con3
+connection con3;
+--echo # Waitng for 'select f1()' to get blocked by a pending MDL lock...
+let $wait_condition=select count(*)=1 from information_schema.processlist
+where state='Waiting for table' and info='select f1()';
 --echo # --> connection default
 connection default;
 commit;
 --echo # --> connection con1
 connection con1;
 --echo # Reaping 'alter procedure p1'...
---reap
+reap;
+--echo # --> connection con2
+connection con2;
+--echo # Reaping 'select f1()'
+reap;
 --echo # --> connection default
 connection default;
 
@@ -269,7 +293,7 @@ select f1();
 --echo # --> connection con1
 connection con1;
 --echo # Sending 'drop function f1'...
---send drop function f1
+send drop function f1;
 --echo # --> connection con2 
 connection con2;
 --echo # Waitng for 'drop function f1' to get blocked on MDL lock...
@@ -277,17 +301,25 @@ let $wait_condition=select count(*)=1 from information_schema.processlist
 where state='Waiting for table' and info='drop function f1';
 --source include/wait_condition.inc
 --echo # Demonstrate that there is a pending exclusive lock.
-lock table t1 read;
---error ER_LOCK_DEADLOCK
-select f1();
-unlock tables;
+--echo # Sending 'select f1()'...
+send select f1();
+--echo # --> connection con3
+connection con3;
+--echo # Waitng for 'select f1()' to get blocked by a pending MDL lock...
+let $wait_condition=select count(*)=1 from information_schema.processlist
+where state='Waiting for table' and info='select f1()';
 --echo # --> connection default
 connection default;
 commit;
 --echo # --> connection con1
 connection con1;
 --echo # Reaping 'drop function f1'...
---reap
+reap;
+--echo # --> connection con2
+connection con2;
+--echo # Reaping 'select f1()'
+--error ER_SP_DOES_NOT_EXIST
+reap;
 --echo # --> connection default
 connection default;
 
@@ -300,7 +332,7 @@ select f1();
 --echo # --> connection con1
 connection con1;
 --echo # Sending 'create function f1'...
---send create function f1() returns int return 2
+send create function f1() returns int return 2;
 --echo # --> connection con2 
 connection con2;
 --echo # Waitng for 'create function f1' to get blocked on MDL lock...
@@ -308,10 +340,13 @@ let $wait_condition=select count(*)=1 from information_schema.processlist
 where state='Waiting for table' and info='create function f1() returns int return 2';
 --source include/wait_condition.inc
 --echo # Demonstrate that there is a pending exclusive lock.
-lock table t1 read;
---error ER_LOCK_DEADLOCK
-select f1();
-unlock tables;
+--echo # Sending 'select f1()'...
+send select f1();
+--echo # --> connection con3
+connection con3;
+--echo # Waitng for 'select f1()' to get blocked by a pending MDL lock...
+let $wait_condition=select count(*)=1 from information_schema.processlist
+where state='Waiting for table' and info='select f1()';
 --echo # --> connection default
 connection default;
 commit;
@@ -319,17 +354,23 @@ commit;
 connection con1;
 --echo # Reaping 'create function f1'...
 --error ER_SP_ALREADY_EXISTS
---reap
+reap;
+--echo # --> connection con2
+connection con2;
+--echo # Reaping 'select f1()'
+reap;
 --echo # --> connection default
 connection default;
+
 --echo # 
 --echo # Test ALTER FUNCTION.
+--echo # 
 begin;
 select f1();
 --echo # --> connection con1
 connection con1;
 --echo # Sending 'alter function f1'...
---send alter function f1 contains sql
+send alter function f1 contains sql;
 --echo # --> connection con2 
 connection con2;
 --echo # Waitng for 'alter function f1' to get blocked on MDL lock...
@@ -337,17 +378,24 @@ let $wait_condition=select count(*)=1 from information_schema.processlist
 where state='Waiting for table' and info='alter function f1 contains sql';
 --source include/wait_condition.inc
 --echo # Demonstrate that there is a pending exclusive lock.
-lock table t1 read;
---error ER_LOCK_DEADLOCK
-select f1();
-unlock tables;
+--echo # Sending 'select f1()'...
+send select f1();
+--echo # --> connection con3
+connection con3;
+--echo # Waitng for 'select f1()' to get blocked by a pending MDL lock...
+let $wait_condition=select count(*)=1 from information_schema.processlist
+where state='Waiting for table' and info='select f1()';
 --echo # --> connection default
 connection default;
 commit;
 --echo # --> connection con1
 connection con1;
 --echo # Reaping 'alter function f1'...
---reap
+reap;
+--echo # --> connection con2
+connection con2;
+--echo # Reaping 'select f1()'
+reap;
 --echo # --> connection default
 connection default;
 drop function f1;
@@ -363,6 +411,7 @@ drop procedure p1;
 --echo #
 create procedure p1() begin end;
 create procedure p2() begin end;
+create table t1 (a int);
 delimiter |;
 create procedure p3()
 begin
@@ -419,7 +468,7 @@ insert into t1 (a) values (1);
 --echo # --> connection con1
 connection con1;
 --echo # Sending 'drop function f1'
---send drop function f1
+send drop function f1;
 --echo # --> connection con2 
 connection con2;
 --echo # Waitng for 'drop function f1' to get blocked on MDL lock...
@@ -432,7 +481,7 @@ commit;
 --echo # --> connection con1
 connection con1;
 --echo # Reaping 'drop function f1'...
---reap
+reap;
 --echo # --> connection default
 connection default;
 --echo #
@@ -445,7 +494,7 @@ select * from v1;
 --echo # --> connection con1
 connection con1;
 --echo # Sending 'drop function f1'
---send drop function f1
+send drop function f1;
 --echo # --> connection con2 
 connection con2;
 --echo # Waitng for 'drop function f1' to get blocked on MDL lock...
@@ -458,7 +507,7 @@ commit;
 --echo # --> connection con1
 connection con1;
 --echo # Reaping 'drop function f1'...
---reap
+reap;
 --echo # --> connection default
 connection default;
 --echo #
@@ -478,7 +527,7 @@ select * from v1;
 --echo # --> connection con1
 connection con1;
 --echo # Sending 'drop procedure p1'
---send drop procedure p1
+send drop procedure p1;
 --echo # --> connection con2 
 connection con2;
 --echo # Waitng for 'drop procedure p1' to get blocked on MDL lock...
@@ -491,7 +540,7 @@ commit;
 --echo # --> connection con1
 connection con1;
 --echo # Reaping 'drop procedure p1'...
---reap
+reap;
 --echo # --> connection default
 connection default;
 
@@ -509,7 +558,7 @@ insert into t1 (a) values (3);
 --echo # --> connection con1
 connection con1;
 --echo # Sending 'drop function f2'
---send drop function f2
+send drop function f2;
 --echo # --> connection con2 
 connection con2;
 --echo # Waitng for 'drop function f2' to get blocked on MDL lock...
@@ -522,7 +571,7 @@ commit;
 --echo # --> connection con1
 connection con1;
 --echo # Reaping 'drop function f2'...
---reap
+reap;
 --echo # --> connection default
 connection default;
 
@@ -536,42 +585,11 @@ drop table t1, t2;
 --echo # acquisition of a shared lock fails during a transaction or
 --echo # we need to back off to flush the sp cache.
 --echo #
---echo # a) A back off due to a lock conflict.
---echo #
-create table t1 (a int);
-create function f1() returns int return 6;
-begin;
-select f1();
---echo # --> connection con1
-connection con1;
---echo # Sending 'drop function f1'...
---send drop function f1
---echo # --> connection con2
-connection con2;
---echo # Waitng for 'drop function f1' to get blocked on MDL lock...
-let $wait_condition=select count(*)=1 from information_schema.processlist
-where state='Waiting for table' and info='drop function f1';
---source include/wait_condition.inc
-begin;
-select * from t1;
---error ER_LOCK_DEADLOCK
-select f1();
-commit;
---echo # --> connection default
-connection default;
-commit;
---echo # --> connection con1
-connection  con1; 
---echo # Reaping 'drop function f1'...
---reap
---echo # --> connection default
-connection default;
---echo #
---echo # b) A back off to flush the cache.
 --echo # Sic: now this situation does not require a back off since we
 --echo # flush the cache on the fly.
 --echo #
 create function f1() returns int return 7;
+create table t1 (a int);
 begin;
 select * from t1;
 # Used to have a back-off here, with optional ER_LOCK_DEADLOCK
@@ -602,7 +620,7 @@ select f2();
 --echo # --> connection con1
 connection con1;
 --echo # Sending 'drop function f1'...
---send drop function f1
+send drop function f1;
 --echo # --> connection con2
 connection con2;
 --echo # Waitng for 'drop function f1' to get blocked on MDL lock...
@@ -610,7 +628,7 @@ let $wait_condition=select count(*)=1 from information_schema.processlist
 where state='Waiting for table' and info='drop function f1';
 --source include/wait_condition.inc
 --echo # Sending 'drop function f2'...
---send drop function f2
+send drop function f2;
 --echo # --> connection default
 connection default;
 --echo # Waitng for 'drop function f2' to get blocked on MDL lock...
@@ -621,14 +639,14 @@ rollback to savepoint sv;
 --echo # --> connection con2
 connection con2;
 --echo # Reaping 'drop function f2'...
---reap
+reap;
 --echo # --> connection default
 connection default;
 unlock tables;
 --echo # --> connection con1
 connection con1;
 --echo # Reaping 'drop function f1'...
---reap
+reap;
 --echo # --> connection default
 connection default;
 --error ER_SP_DOES_NOT_EXIST
@@ -678,7 +696,7 @@ select f1();
 --echo # --> connection con1
 connection con1;
 --echo # Sending 'alter function f1 ...'...
---send alter function f1 comment "comment"
+send alter function f1 comment "comment";
 --echo # --> connection con2
 connection con2;
 --echo # Waitng for 'alter function f1 ...' to get blocked on MDL lock...
@@ -686,7 +704,7 @@ let $wait_condition=select count(*)=1 from information_schema.processlist
 where state='Waiting for table' and info like 'alter function f1 comment%';
 --source include/wait_condition.inc
 --echo # Sending 'call p1()'...
---send call p1()
+send call p1();
 connection default;
 --echo # Waitng for 'call p1()' to get blocked on MDL lock on f1...
 let $wait_condition=select count(*)=1 from information_schema.processlist
@@ -697,11 +715,11 @@ commit;
 --echo # --> connection con1
 connection con1;
 --echo # Reaping 'alter function f1 ...'
---reap
+reap;
 --echo # --> connection con2
 connection con2;
 --echo # Reaping 'call p1()'...
---reap
+reap;
 deallocate prepare stmt;
 --echo # --> connection default
 connection default;
@@ -725,7 +743,7 @@ select f1();
 --echo # --> connection con1
 connection con1;
 --echo # Sending 'alter function f1 ...'...
---send alter function f1 comment "comment"
+send alter function f1 comment "comment";
 --echo # --> connection con2
 connection con2;
 --echo # Waitng for 'alter function f1 ...' to get blocked on MDL lock...
@@ -754,7 +772,7 @@ begin
 end|
 delimiter ;|
 --echo # Sending 'call p1()'...
---send call p1()
+send call p1();
 connection default;
 --echo # Waitng for 'call p1()' to get blocked on MDL lock on f1...
 let $wait_condition=select count(*)=1 from information_schema.processlist
@@ -765,11 +783,11 @@ commit;
 --echo # --> connection con1
 connection con1;
 --echo # Reaping 'alter function f1 ...'
---reap
+reap;
 --echo # --> connection con2
 connection con2;
 --echo # Reaping 'call p1()'...
---reap
+reap;
 --echo # --> connection default
 connection default;
 drop function f1;
@@ -804,7 +822,7 @@ select get_lock("30977", 0);
 --echo # --> connection default
 connection default;
 --echo # Sending 'select f3()'...
---send select f3()
+send select f3();
 --echo # --> connection con1 
 connection con1;
 --echo # Waitng for 'select f3()' to get blocked on the user level lock...
@@ -819,7 +837,7 @@ select release_lock("30977");
 connection default;
 --echo # Reaping 'select f3()'...
 --echo # Routine 'f2()' should exist and get executed successfully.
---reap
+reap;
 select @var;
 drop function f1;
 drop function f2;
@@ -869,6 +887,10 @@ disconnect con1;
 connection con2;
 disconnect con2;
 --source include/wait_until_disconnected.inc
+--echo # --> connection con3
+connection con3;
+disconnect con3;
+--source include/wait_until_disconnected.inc
 --echo # --> connection default
 connection default;
 --echo #

sql/log_event_old.cc

@@ -1506,7 +1506,7 @@ int Old_rows_log_event::do_apply_event(Relay_log_info const *rli)
        */
       thd->binlog_flush_pending_rows_event(false);
       TABLE_LIST *tables= rli->tables_to_lock;
-      close_tables_for_reopen(thd, &tables);
+      close_tables_for_reopen(thd, &tables, NULL);
 
       uint tables_count= rli->tables_to_lock_count;
       if ((error= open_tables(thd, &tables, &tables_count, 0)))

sql/mdl.cc

@@ -196,6 +196,7 @@ void MDL_context::init(THD *thd_arg)
     to empty the list.
   */
   m_tickets.empty();
+  m_is_waiting_in_mdl= FALSE;
 }
 
 
@@ -803,13 +804,27 @@ MDL_context::clone_ticket(MDL_request *mdl_request)
   @retval FALSE  Lock is not a shared one or no thread was woken up
 */
 
-static bool notify_shared_lock(THD *thd, MDL_ticket *conflicting_ticket)
+bool notify_shared_lock(THD *thd, MDL_ticket *conflicting_ticket)
 {
   bool woke= FALSE;
   if (conflicting_ticket->is_shared())
   {
     THD *conflicting_thd= conflicting_ticket->get_ctx()->get_thd();
     DBUG_ASSERT(thd != conflicting_thd); /* Self-deadlock */
+
+    /*
+      If the thread that holds the conflicting lock is waiting
+      on an MDL lock, wake it up by broadcasting on COND_mdl.
+      Otherwise it must be waiting on a table-level lock
+      or some other non-MDL resource, so delegate its waking up
+      to an external call.
+    */
+    if (conflicting_ticket->get_ctx()->is_waiting_in_mdl())
+    {
+      pthread_cond_broadcast(&COND_mdl);
+      woke= TRUE;
+    }
+    else
       woke= mysql_notify_thread_having_shared_lock(thd, conflicting_thd);
   }
   return woke;
@@ -957,7 +972,7 @@ bool MDL_context::acquire_exclusive_locks(MDL_request_list *mdl_requests)
         to abort this thread once again.
       */
       struct timespec abstime;
-      set_timespec(abstime, 10);
+      set_timespec(abstime, 1);
       pthread_cond_timedwait(&COND_mdl, &LOCK_mdl, &abstime);
     }
     if (mysys_var->abort)
@@ -1032,6 +1047,7 @@ MDL_ticket::upgrade_shared_lock_to_exclusive()
   const char *old_msg;
   st_my_thread_var *mysys_var= my_thread_var;
   THD *thd= m_ctx->get_thd();
+  MDL_ticket *pending_ticket;
 
   DBUG_ENTER("MDL_ticket::upgrade_shared_lock_to_exclusive");
   DEBUG_SYNC(thd, "mdl_upgrade_shared_lock_to_exclusive");
@@ -1045,8 +1061,22 @@ MDL_ticket::upgrade_shared_lock_to_exclusive()
   /* Only allow upgrades from MDL_SHARED_UPGRADABLE */
   DBUG_ASSERT(m_type == MDL_SHARED_UPGRADABLE);
 
+  /*
+    Create an auxiliary ticket to represent a pending exclusive
+    lock and add it to the 'waiting' queue for the duration
+    of upgrade. During upgrade we abort waits of connections
+    that own conflicting locks. A pending request is used
+    to signal such connections that upon waking up they
+    must back off, rather than fall into sleep again.
+  */
+  if (! (pending_ticket= MDL_ticket::create(m_ctx, MDL_EXCLUSIVE)))
+    DBUG_RETURN(TRUE);
+
   pthread_mutex_lock(&LOCK_mdl);
 
+  pending_ticket->m_lock= m_lock;
+  m_lock->waiting.push_front(pending_ticket);
+
   old_msg= MDL_ENTER_COND(thd, mysys_var);
 
   /*
@@ -1088,6 +1118,30 @@ MDL_ticket::upgrade_shared_lock_to_exclusive()
     MDL_ticket *conflicting_ticket;
     MDL_lock::Ticket_iterator it(m_lock->granted);
 
+    /*
+      A temporary work-around to avoid deadlocks/livelocks in
+      a situation when in one connection ALTER TABLE tries to
+      upgrade its metadata lock and in another connection
+      the active transaction already got this lock in some
+      of its earlier statements.
+      In such case this transaction always succeeds with getting
+      a metadata lock on the table -- it already has one.
+      But later on it may block on the table level lock, since ALTER
+      got TL_WRITE_ALLOW_READ, and subsequently get aborted
+      by notify_shared_lock().
+      An abort will lead to a back off, and a second attempt to
+      get an MDL lock (successful), and a table lock (-> livelock).
+
+      The call below breaks this loop by forcing transactions to call
+      tdc_wait_for_old_versions() (even if the transaction doesn't need
+      any new metadata locks), which in turn will check if someone
+      is waiting on the owned MDL lock, and produce ER_LOCK_DEADLOCK.
+
+      TODO: Long-term such deadlocks/livelock will be resolved within
+            MDL subsystem and thus this call will become unnecessary.
+    */
+    mysql_abort_transactions_with_shared_lock(&m_lock->key);
+
     while ((conflicting_ticket= it++))
     {
       if (conflicting_ticket->m_ctx != m_ctx)
@@ -1108,12 +1162,15 @@ MDL_ticket::upgrade_shared_lock_to_exclusive()
         to abort this thread once again.
       */
       struct timespec abstime;
-      set_timespec(abstime, 10);
+      set_timespec(abstime, 1);
       DBUG_PRINT("info", ("Failed to wake-up from table-level lock ... sleeping"));
       pthread_cond_timedwait(&COND_mdl, &LOCK_mdl, &abstime);
     }
     if (mysys_var->abort)
     {
+      /* Remove and destroy the auxiliary pending ticket. */
+      m_lock->waiting.remove(pending_ticket);
+      MDL_ticket::destroy(pending_ticket);
       /* Pending requests for shared locks can be satisfied now. */
       pthread_cond_broadcast(&COND_mdl);
       MDL_EXIT_COND(thd, mysys_var, old_msg);
@@ -1124,6 +1181,11 @@ MDL_ticket::upgrade_shared_lock_to_exclusive()
   m_lock->type= MDL_lock::MDL_LOCK_EXCLUSIVE;
   /* Set the new type of lock in the ticket. */
   m_type= MDL_EXCLUSIVE;
+
+  /* Remove and destroy the auxiliary pending ticket. */
+  m_lock->waiting.remove(pending_ticket);
+  MDL_ticket::destroy(pending_ticket);
+
   if (m_lock->cached_object)
     (*m_lock->cached_object_release_hook)(m_lock->cached_object);
   m_lock->cached_object= 0;
@@ -1239,6 +1301,59 @@ bool MDL_context::acquire_global_shared_lock()
 }
 
 
+/**
+  Check if there are any pending exclusive locks which conflict
+  with shared locks held by this thread.
+
+  @pre The caller already has acquired LOCK_mdl.
+
+  @return TRUE   If there are any pending conflicting locks.
+          FALSE  Otherwise.
+*/
+
+bool MDL_context::can_wait_lead_to_deadlock_impl() const
+{
+  Ticket_iterator ticket_it(m_tickets);
+  MDL_ticket *ticket;
+
+  while ((ticket= ticket_it++))
+  {
+    /*
+      In MySQL we never call this method while holding exclusive or
+      upgradeable shared metadata locks.
+      Otherwise we would also have to check for the presence of pending
+      requests for conflicting types of global lock.
+      In addition MDL_ticket::has_pending_conflicting_lock_impl()
+      won't work properly for exclusive type of lock.
+    */
+    DBUG_ASSERT(! ticket->is_upgradable_or_exclusive());
+
+    if (ticket->has_pending_conflicting_lock_impl())
+      return TRUE;
+  }
+  return FALSE;
+}
+
+
+/**
+  Implement a simple deadlock detection heuristic: check if there
+  are any pending exclusive locks which conflict with shared locks
+  held by this thread. In that case waiting can be circular,
+  i.e. lead to a deadlock.
+
+  @return TRUE if there are any conflicting locks, FALSE otherwise.
+*/
+
+bool MDL_context::can_wait_lead_to_deadlock() const
+{
+  bool result;
+  pthread_mutex_lock(&LOCK_mdl);
+  result= can_wait_lead_to_deadlock_impl();
+  pthread_mutex_unlock(&LOCK_mdl);
+  return result;
+}
+
+
 /**
   Wait until there will be no locks that conflict with lock requests
   in the given list.
@@ -1249,7 +1364,7 @@ bool MDL_context::acquire_global_shared_lock()
   Does not acquire the locks!
 
   @retval FALSE  Success. One can try to obtain metadata locks.
-  @retval TRUE   Failure (thread was killed)
+  @retval TRUE   Failure (thread was killed or deadlock is possible).
 */
 
 bool
@@ -1278,6 +1393,26 @@ MDL_context::wait_for_locks(MDL_request_list *mdl_requests)
     mysql_ha_flush(m_thd);
     pthread_mutex_lock(&LOCK_mdl);
     old_msg= MDL_ENTER_COND(m_thd, mysys_var);
+
+    /*
+      In cases when we wait while still holding some metadata
+      locks deadlocks are possible.
+      To avoid them we use the following simple empiric - don't
+      wait for new lock request to be satisfied if for one of the
+      locks which are already held by this connection there is
+      a conflicting request (i.e. this connection should not wait
+      if someone waits for it).
+      This empiric should work well (e.g. give low number of false
+      negatives) in situations when conflicts are rare (in our
+      case this is true since DDL statements should be rare).
+    */
+    if (can_wait_lead_to_deadlock_impl())
+    {
+      MDL_EXIT_COND(m_thd, mysys_var, old_msg);
+      my_error(ER_LOCK_DEADLOCK, MYF(0));
+      return TRUE;
+    }
+
     it.rewind();
     while ((mdl_request= it++))
     {
@@ -1301,7 +1436,9 @@ MDL_context::wait_for_locks(MDL_request_list *mdl_requests)
       MDL_EXIT_COND(m_thd, mysys_var, old_msg);
       break;
     }
+    m_is_waiting_in_mdl= TRUE;
     pthread_cond_wait(&COND_mdl, &LOCK_mdl);
+    m_is_waiting_in_mdl= FALSE;
     /* As a side-effect MDL_EXIT_COND() unlocks LOCK_mdl. */
     MDL_EXIT_COND(m_thd, mysys_var, old_msg);
   }
@@ -1550,21 +1687,38 @@ MDL_context::is_lock_owner(MDL_key::enum_mdl_namespace mdl_namespace,
   existing shared lock.
 
   @pre The ticket must match an acquired lock.
+  @pre The caller already has acquired LOCK_mdl.
 
-  @param ticket Shared lock against which check should be performed.
+  @return TRUE if there is a conflicting lock request, FALSE otherwise.
+*/
 
-  @return TRUE if there are any conflicting locks, FALSE otherwise.
+bool MDL_ticket::has_pending_conflicting_lock_impl() const
+{
+  DBUG_ASSERT(is_shared());
+  safe_mutex_assert_owner(&LOCK_mdl);
+
+  return !m_lock->waiting.is_empty();
+}
+
+
+/**
+  Check if we have any pending exclusive locks which conflict with
+  existing shared lock.
+
+  @pre The ticket must match an acquired lock.
+
+  @return TRUE if there is a pending conflicting lock request,
+          FALSE otherwise.
 */
 
 bool MDL_ticket::has_pending_conflicting_lock() const
 {
   bool result;
 
-  DBUG_ASSERT(is_shared());
   safe_mutex_assert_not_owner(&LOCK_open);
 
   pthread_mutex_lock(&LOCK_mdl);
-  result= !m_lock->waiting.is_empty();
+  result= has_pending_conflicting_lock_impl();
   pthread_mutex_unlock(&LOCK_mdl);
   return result;
 }

sql/mdl.h

@@ -300,6 +300,8 @@ class MDL_ticket
 private:
   MDL_ticket(const MDL_ticket &);               /* not implemented */
   MDL_ticket &operator=(const MDL_ticket &);    /* not implemented */
+
+  bool has_pending_conflicting_lock_impl() const;
 };
 
 
@@ -380,10 +382,19 @@ class MDL_context
   void release_transactional_locks();
   void rollback_to_savepoint(MDL_ticket *mdl_savepoint);
 
+  bool can_wait_lead_to_deadlock() const;
+
   inline THD *get_thd() const { return m_thd; }
+  
+  bool is_waiting_in_mdl() const { return m_is_waiting_in_mdl; }
 private:
   Ticket_list m_tickets;
   bool m_has_global_shared_lock;
+  /**
+    Indicates that the owner of this context is waiting in
+    wait_for_locks() method.
+  */
+  bool m_is_waiting_in_mdl;
   /**
     This member has two uses:
     1) When entering LOCK TABLES mode, remember the last taken
@@ -397,6 +408,7 @@ class MDL_context
   THD *m_thd;
 private:
   void release_ticket(MDL_ticket *ticket);
+  bool can_wait_lead_to_deadlock_impl() const;
   MDL_ticket *find_ticket(MDL_request *mdl_req,
                           bool *is_lt_or_ha);
   void release_locks_stored_before(MDL_ticket *sentinel);
@@ -413,6 +425,7 @@ void mdl_destroy();
 
 extern bool mysql_notify_thread_having_shared_lock(THD *thd, THD *in_use);
 extern void mysql_ha_flush(THD *thd);
+extern void mysql_abort_transactions_with_shared_lock(const MDL_key *mdl_key);
 extern "C" const char *set_thd_proc_info(THD *thd, const char *info,
                                          const char *calling_function,
                                          const char *calling_file,

sql/mysql_priv.h

@@ -1522,7 +1522,8 @@ void free_io_cache(TABLE *entry);
 void intern_close_table(TABLE *entry);
 bool close_thread_table(THD *thd, TABLE **table_ptr);
 void close_temporary_tables(THD *thd);
-void close_tables_for_reopen(THD *thd, TABLE_LIST **tables);
+void close_tables_for_reopen(THD *thd, TABLE_LIST **tables,
+                             MDL_ticket *mdl_savepoint);
 TABLE_LIST *find_table_in_list(TABLE_LIST *table,
                                TABLE_LIST *TABLE_LIST::*link,
                                const char *db_name,

sql/sql_base.cc

@@ -2142,25 +2142,13 @@ bool rename_temporary_table(THD* thd, TABLE *table, const char *db,
 bool wait_while_table_is_used(THD *thd, TABLE *table,
                               enum ha_extra_function function)
 {
-  enum thr_lock_type old_lock_type;
   DBUG_ENTER("wait_while_table_is_used");
   DBUG_PRINT("enter", ("table: '%s'  share: 0x%lx  db_stat: %u  version: %lu",
                        table->s->table_name.str, (ulong) table->s,
                        table->db_stat, table->s->version));
 
-  /* Ensure no one can reopen table before it's removed */
-  pthread_mutex_lock(&LOCK_open);
-  table->s->version= 0;
-  pthread_mutex_unlock(&LOCK_open);
-
-  old_lock_type= table->reginfo.lock_type;
-  mysql_lock_abort(thd, table, TRUE);	/* end threads waiting on lock */
-
   if (table->mdl_ticket->upgrade_shared_lock_to_exclusive())
-  {
-    mysql_lock_downgrade_write(thd, table, old_lock_type);
     DBUG_RETURN(TRUE);
-  }
 
   pthread_mutex_lock(&LOCK_open);
   tdc_remove_table(thd, TDC_RT_REMOVE_NOT_OWN,
@@ -3722,8 +3710,9 @@ static bool auto_repair_table(THD *thd, TABLE_LIST *table_list)
 
 Open_table_context::Open_table_context(THD *thd)
   :m_action(OT_NO_ACTION),
-  m_can_deadlock((thd->in_multi_stmt_transaction() ||
-                 thd->mdl_context.lt_or_ha_sentinel())&&
+   m_start_of_statement_svp(thd->mdl_context.mdl_savepoint()),
+   m_has_locks((thd->in_multi_stmt_transaction() ||
+                thd->mdl_context.lt_or_ha_sentinel()) &&
                thd->mdl_context.has_locks())
 {}
 
@@ -3741,12 +3730,22 @@ Open_table_context::
 request_backoff_action(enum_open_table_action action_arg)
 {
   /*
-    We have met a exclusive metadata lock or a old version of
-    table and we are inside a transaction that already hold locks.
-    We can't follow the locking protocol in this scenario as it
-    might lead to deadlocks.
+    We are inside a transaction that already holds locks and have
+    met a broken table or a table which needs re-discovery.
+    Performing any recovery action requires acquiring an exclusive
+    metadata lock on this table. Doing that with locks breaks the
+    metadata locking protocol and might lead to deadlocks,
+    so we report an error.
+
+    However, if we have only met a conflicting lock or an old
+    TABLE version, and just need to wait for the conflict to
+    disappear/old version to go away, allow waiting.
+    While waiting, we use a simple empiric to detect
+    deadlocks: we never wait on someone who's waiting too.
+    Waiting will be done after releasing metadata locks acquired
+    by this statement.
   */
-  if (m_can_deadlock)
+  if (m_has_locks && action_arg != OT_WAIT)
   {
     my_error(ER_LOCK_DEADLOCK, MYF(0));
     return TRUE;
@@ -4364,7 +4363,7 @@ bool open_tables(THD *thd, TABLE_LIST **start, uint *counter, uint flags,
             elements from the table list (if MERGE tables are involved),
           */
           TABLE_LIST *failed_table= *table_to_open;
-          close_tables_for_reopen(thd, start);
+          close_tables_for_reopen(thd, start, ot_ctx.start_of_statement_svp());
 
           /*
             Here we rely on the fact that 'tables' still points to the valid
@@ -4414,7 +4413,8 @@ bool open_tables(THD *thd, TABLE_LIST **start, uint *counter, uint flags,
         {
           if (ot_ctx.can_recover_from_failed_open())
           {
-            close_tables_for_reopen(thd, start);
+            close_tables_for_reopen(thd, start,
+                                    ot_ctx.start_of_statement_svp());
             if (ot_ctx.recover_from_failed_open(thd, &rt->mdl_request, NULL))
               goto err;
 
@@ -4827,14 +4827,14 @@ TABLE *open_ltable(THD *thd, TABLE_LIST *table_list, thr_lock_type lock_type,
   while ((error= open_table(thd, table_list, thd->mem_root, &ot_ctx, 0)) &&
          ot_ctx.can_recover_from_failed_open())
   {
-    /* We can't back off with an open HANDLER, we don't wait with locks. */
+    /* We never have an open HANDLER or LOCK TABLES here. */
     DBUG_ASSERT(thd->mdl_context.lt_or_ha_sentinel() == NULL);
     /*
       Even though we have failed to open table we still need to
       call release_transactional_locks() to release metadata locks which
       might have been acquired successfully.
     */
-    thd->mdl_context.release_transactional_locks();
+    thd->mdl_context.rollback_to_savepoint(ot_ctx.start_of_statement_svp());
     table_list->mdl_request.ticket= 0;
     if (ot_ctx.recover_from_failed_open(thd, &table_list->mdl_request,
                                         table_list))
@@ -4875,26 +4875,15 @@ TABLE *open_ltable(THD *thd, TABLE_LIST *table_list, thr_lock_type lock_type,
                                             lock_flags, &refresh)))
         {
           if (refresh)
-          {
-            if (ot_ctx.can_deadlock())
-            {
-              my_error(ER_LOCK_DEADLOCK, MYF(0));
-              table= 0;
-            }
-            else
           {
             close_thread_tables(thd);
             table_list->table= NULL;
             table_list->mdl_request.ticket= NULL;
-              /*
-                We can't back off with an open HANDLER,
-                we don't wait with locks.
-              */
+            /* We never have an open HANDLER or LOCK TABLES here. */
             DBUG_ASSERT(thd->mdl_context.lt_or_ha_sentinel() == NULL);
-              thd->mdl_context.release_transactional_locks();
+            thd->mdl_context.rollback_to_savepoint(ot_ctx.start_of_statement_svp());
             goto retry;
           }
-          }
           else
             table= 0;
         }
@@ -4941,7 +4930,16 @@ bool open_and_lock_tables_derived(THD *thd, TABLE_LIST *tables,
 {
   uint counter;
   bool need_reopen;
-  bool has_locks= thd->mdl_context.has_locks();
+  /*
+    Remember the set of metadata locks which this connection
+    managed to acquire before the start of the current statement.
+    It can be either transaction-scope locks, or HANDLER locks,
+    or LOCK TABLES locks. If mysql_lock_tables() fails with
+    need_reopen request, we'll use it to instruct
+    close_tables_for_reopen() to release all locks of this
+    statement.
+  */
+  MDL_ticket *start_of_statement_svp= thd->mdl_context.mdl_savepoint();
   DBUG_ENTER("open_and_lock_tables_derived");
   DBUG_PRINT("enter", ("derived handling: %d", derived));
 
@@ -4960,13 +4958,7 @@ bool open_and_lock_tables_derived(THD *thd, TABLE_LIST *tables,
       break;
     if (!need_reopen)
       DBUG_RETURN(TRUE);
-    if ((thd->in_multi_stmt_transaction() ||
-         thd->mdl_context.lt_or_ha_sentinel()) && has_locks)
-    {
-      my_error(ER_LOCK_DEADLOCK, MYF(0));
-      DBUG_RETURN(TRUE);
-    }
-    close_tables_for_reopen(thd, &tables);
+    close_tables_for_reopen(thd, &tables, start_of_statement_svp);
   }
   if (derived &&
       (mysql_handle_derived(thd->lex, &mysql_derived_prepare) ||
@@ -5280,6 +5272,8 @@ bool lock_tables(THD *thd, TABLE_LIST *tables, uint count,
                                         flags, need_reopen)))
       DBUG_RETURN(TRUE);
 
+    DEBUG_SYNC(thd, "after_lock_tables_takes_lock");
+
     if (thd->lex->requires_prelocking() &&
         thd->lex->sql_command != SQLCOM_LOCK_TABLES)
     {
@@ -5379,18 +5373,24 @@ bool lock_tables(THD *thd, TABLE_LIST *tables, uint count,
 }
 
 
-/*
+/**
   Prepare statement for reopening of tables and recalculation of set of
   prelocked tables.
 
-  SYNOPSIS
-    close_tables_for_reopen()
-      thd    in     Thread context
-      tables in/out List of tables which we were trying to open and lock
-
+  @param[in] thd         Thread context.
+  @param[in,out] tables  List of tables which we were trying to open
+                         and lock.
+  @param[in] start_of_statement_svp MDL savepoint which represents the set
+                         of metadata locks which the current transaction
+                         managed to acquire before execution of the current
+                         statement and to which we should revert before
+                         trying to reopen tables. NULL if no metadata locks
+                         were held and thus all metadata locks should be
+                         released.
 */
 
-void close_tables_for_reopen(THD *thd, TABLE_LIST **tables)
+void close_tables_for_reopen(THD *thd, TABLE_LIST **tables,
+                             MDL_ticket *start_of_statement_svp)
 {
   TABLE_LIST *first_not_own_table= thd->lex->first_not_own_table();
   TABLE_LIST *tmp;
@@ -5425,13 +5425,7 @@ void close_tables_for_reopen(THD *thd, TABLE_LIST **tables)
   for (tmp= first_not_own_table; tmp; tmp= tmp->next_global)
     tmp->mdl_request.ticket= NULL;
   close_thread_tables(thd);
-  /* We can't back off with an open HANDLERs, we must not wait with locks. */
-  DBUG_ASSERT(thd->mdl_context.lt_or_ha_sentinel() == NULL);
-  /*
-    Due to the above assert, this effectively releases *all* locks
-    of this session, so that we can safely wait on tables.
-  */
-  thd->mdl_context.release_transactional_locks();
+  thd->mdl_context.rollback_to_savepoint(start_of_statement_svp);
 }
 
 
@@ -8413,6 +8407,8 @@ bool mysql_notify_thread_having_shared_lock(THD *thd, THD *in_use)
     But in case a thread has an open HANDLER statement,
     (and thus already grabbed a metadata lock), it gets
     blocked only too late -- at the table cache level.
+    Starting from 5.5, this could also easily happen in
+    a multi-statement transaction.
   */
   broadcast_refresh();
   pthread_mutex_unlock(&LOCK_open);
@@ -8420,6 +8416,28 @@ bool mysql_notify_thread_having_shared_lock(THD *thd, THD *in_use)
 }
 
 
+/**
+  Force transactions holding shared metadata lock on the table to call
+  MDL_context::can_wait_lead_to_deadlock() even if they don't need any
+  new metadata locks so they can detect potential deadlocks between
+  metadata locking subsystem and table-level locks.
+
+  @param mdl_key MDL key for the table on which we are upgrading
+                 metadata lock.
+*/
+
+void mysql_abort_transactions_with_shared_lock(const MDL_key *mdl_key)
+{
+  if (mdl_key->mdl_namespace() == MDL_key::TABLE)
+  {
+    pthread_mutex_lock(&LOCK_open);
+    tdc_remove_table(NULL, TDC_RT_REMOVE_UNUSED, mdl_key->db_name(),
+                     mdl_key->name());
+    pthread_mutex_unlock(&LOCK_open);
+  }
+}
+
+
 /**
    Remove all or some (depending on parameter) instances of TABLE and
    TABLE_SHARE from the table definition cache.
@@ -8525,6 +8543,25 @@ tdc_wait_for_old_versions(THD *thd, MDL_request_list *mdl_requests)
             to broadcast on COND_refresh because of this.
     */
     mysql_ha_flush(thd);
+
+    /*
+      Check if there is someone waiting for one of metadata locks
+      held by this connection and return an error if that's the
+      case, since this situation may lead to a deadlock.
+      This can happen, when, for example, this connection is
+      waiting for an old version of some table to go away and
+      another connection is trying to upgrade its shared
+      metadata lock to exclusive, and thus is waiting
+      for this to release its lock. We must check for
+      the condition on each iteration of the loop to remove
+      any window for a race.
+    */
+    if (thd->mdl_context.can_wait_lead_to_deadlock())
+    {
+      my_error(ER_LOCK_DEADLOCK, MYF(0));
+      return TRUE;
+    }
+
     pthread_mutex_lock(&LOCK_open);
 
     MDL_request_list::Iterator it(*mdl_requests);

sql/sql_class.h

@@ -1297,18 +1297,29 @@ class Open_table_context
 
   bool can_recover_from_failed_open() const
   { return m_action != OT_NO_ACTION; }
-  bool can_deadlock() const { return m_can_deadlock; }
+
+  /**
+    When doing a back-off, we close all tables acquired by this
+    statement.  Return an MDL savepoint taken at the beginning of
+    the statement, so that we can rollback to it before waiting on
+    locks.
+  */
+  MDL_ticket *start_of_statement_svp() const
+  {
+    return m_start_of_statement_svp;
+  }
 private:
   /** List of requests for all locks taken so far. Used for waiting on locks. */
   MDL_request_list m_mdl_requests;
   /** Back off action. */
   enum enum_open_table_action m_action;
+  MDL_ticket *m_start_of_statement_svp;
   /**
     Whether we had any locks when this context was created.
     If we did, they are from the previous statement of a transaction,
     and we can't safely do back-off (and release them).
   */
-  bool m_can_deadlock;
+  bool m_has_locks;
 };
 
 

sql/sql_insert.cc

@@ -2530,7 +2530,7 @@ pthread_handler_t handle_delayed_insert(void *arg)
               aborted. Try to reopen table and if it fails die.
             */
             TABLE_LIST *tl_ptr = &di->table_list;
-            close_tables_for_reopen(thd, &tl_ptr);
+            close_tables_for_reopen(thd, &tl_ptr, NULL);
             di->table= 0;
             if (di->open_and_lock_table())
             {

sql/sql_plist.h

@@ -132,11 +132,11 @@ class I_P_List
 template <typename T, typename B>
 class I_P_List_iterator
 {
-  I_P_List<T, B> *list;
+  const I_P_List<T, B> *list;
   T *current;
 public:
-  I_P_List_iterator(I_P_List<T, B> &a) : list(&a), current(a.first) {}
-  I_P_List_iterator(I_P_List<T, B> &a, T* current_arg) : list(&a), current(current_arg) {}
+  I_P_List_iterator(const I_P_List<T, B> &a) : list(&a), current(a.first) {}
+  I_P_List_iterator(const I_P_List<T, B> &a, T* current_arg) : list(&a), current(current_arg) {}
   inline void init(I_P_List<T, B> &a)
   {
     list= &a;

sql/sql_show.cc

@@ -2941,7 +2941,7 @@ fill_schema_show_cols_or_idxs(THD *thd, TABLE_LIST *tables,
                                            table, res, db_name,
                                            table_name));
    thd->temporary_tables= 0;
-   close_tables_for_reopen(thd, &show_table_list);
+   close_tables_for_reopen(thd, &show_table_list, NULL);
    DBUG_RETURN(error);
 }
 
@@ -3500,7 +3500,7 @@ int get_all_tables(THD *thd, TABLE_LIST *tables, COND *cond)
               res= schema_table->process_table(thd, show_table_list, table,
                                                res, &orig_db_name,
                                                &tmp_lex_string);
-              close_tables_for_reopen(thd, &show_table_list);
+              close_tables_for_reopen(thd, &show_table_list, NULL);
             }
             DBUG_ASSERT(!lex->query_tables_own_last);
             if (res)

sql/sql_update.cc

@@ -206,6 +206,7 @@ int mysql_update(THD *thd,
   ulonglong     id;
   List<Item> all_fields;
   THD::killed_state killed_status= THD::NOT_KILLED;
+  MDL_ticket *start_of_statement_svp= thd->mdl_context.mdl_savepoint();
   DBUG_ENTER("mysql_update");
 
   for ( ; ; )
@@ -226,7 +227,7 @@ int mysql_update(THD *thd,
       break;
     if (!need_reopen)
       DBUG_RETURN(1);
-    close_tables_for_reopen(thd, &table_list);
+    close_tables_for_reopen(thd, &table_list, start_of_statement_svp);
   }
 
   if (mysql_handle_derived(thd->lex, &mysql_derived_prepare) ||
@@ -981,6 +982,7 @@ int mysql_multi_update_prepare(THD *thd)
   const bool using_lock_tables= thd->locked_tables_mode != LTM_NONE;
   bool original_multiupdate= (thd->lex->sql_command == SQLCOM_UPDATE_MULTI);
   bool need_reopen= FALSE;
+  MDL_ticket *start_of_statement_svp= thd->mdl_context.mdl_savepoint();
   DBUG_ENTER("mysql_multi_update_prepare");
 
   /* following need for prepared statements, to run next time multi-update */
@@ -1145,7 +1147,7 @@ int mysql_multi_update_prepare(THD *thd)
     */
     cleanup_items(thd->free_list);
 
-    close_tables_for_reopen(thd, &table_list);
+    close_tables_for_reopen(thd, &table_list, start_of_statement_svp);
     goto reopen_tables;
   }