NULL MERGE UP to 5.1.

Apply innodb-5.0-* snapshots:  ss1489 and ss1547.

Fixes:
Bug#9709:  InnoDB inconsistensy causes "Operating System Error 32/33"
Bug#22819: SHOW INNODB STATUS crashes the server with an assertion failure under high load
Bug#25645: Assertion failure in file srv0srv.c
Bug#27294: insert into ... select ... causes crash with innodb_locks_unsafe_for_binlog=1
Bug#28138: indexing column prefixes produces corruption in InnoDB


innobase/btr/btr0btr.c:
  Apply innodb-5.0-* snapshots:  ss1489 and ss1547.
  
  Revision r1547:
  branches/5.0: Merge r1546 from trunk:
  
  When buffering an insert to a prefix index of a variable-length column,
  do not incorrectly mark the column as fixed-length.  (Bug #28138)
  
  ibuf_entry_build(): Instead of prefix_len, pass fixed_len to
  dtype_new_store_for_order_and_null_size().  Add debug assertions.
  
  btr_index_rec_validate(): Correct a comment about prefix indexes.
  
  rec_get_converted_size_new(), rec_convert_dtuple_to_rec_new(): Add
  debug assertions and comments.
  
  dict_col_type_assert_equal(): New debug function.
innobase/buf/buf0buf.c:
  Apply innodb-5.0-* snapshots:  ss1489 and ss1547.
  
  Revision r1502:
  branches/5.0: Add debug code for Bug 26081. This change has some debug
  assertions that have been promoted to normal assertions. These will need
  to be undone once we've nailed this bug.
innobase/buf/buf0lru.c:
  Apply innodb-5.0-* snapshots:  ss1489 and ss1547.
  
  Revision r1502:
  branches/5.0: Add debug code for Bug 26081. This change has some debug
  assertions that have been promoted to normal assertions. These will need
  to be undone once we've nailed this bug.
innobase/ibuf/ibuf0ibuf.c:
  Apply innodb-5.0-* snapshots:  ss1489 and ss1547.
  
  Revision r1547:
  branches/5.0: Merge r1546 from trunk:
  
  When buffering an insert to a prefix index of a variable-length column,
  do not incorrectly mark the column as fixed-length.  (Bug #28138)
  
  ibuf_entry_build(): Instead of prefix_len, pass fixed_len to
  dtype_new_store_for_order_and_null_size().  Add debug assertions.
  
  btr_index_rec_validate(): Correct a comment about prefix indexes.
  
  rec_get_converted_size_new(), rec_convert_dtuple_to_rec_new(): Add
  debug assertions and comments.
  
  dict_col_type_assert_equal(): New debug function.
innobase/include/buf0buf.ic:
  Apply innodb-5.0-* snapshots:  ss1489 and ss1547.
  
  Revision r1502:
  branches/5.0: Add debug code for Bug 26081. This change has some debug
  assertions that have been promoted to normal assertions. These will need
  to be undone once we've nailed this bug.
innobase/include/buf0lru.h:
  Apply innodb-5.0-* snapshots:  ss1489 and ss1547.
  
  Revision r1502:
  branches/5.0: Add debug code for Bug 26081. This change has some debug
  assertions that have been promoted to normal assertions. These will need
  to be undone once we've nailed this bug.
innobase/include/dict0dict.h:
  Apply innodb-5.0-* snapshots:  ss1489 and ss1547.
  
  Revision r1547:
  branches/5.0: Merge r1546 from trunk:
  
  When buffering an insert to a prefix index of a variable-length column,
  do not incorrectly mark the column as fixed-length.  (Bug #28138)
  
  ibuf_entry_build(): Instead of prefix_len, pass fixed_len to
  dtype_new_store_for_order_and_null_size().  Add debug assertions.
  
  btr_index_rec_validate(): Correct a comment about prefix indexes.
  
  rec_get_converted_size_new(), rec_convert_dtuple_to_rec_new(): Add
  debug assertions and comments.
  
  dict_col_type_assert_equal(): New debug function.
innobase/include/dict0dict.ic:
  Apply innodb-5.0-* snapshots:  ss1489 and ss1547.
  
  Revision r1547:
  branches/5.0: Merge r1546 from trunk:
  
  When buffering an insert to a prefix index of a variable-length column,
  do not incorrectly mark the column as fixed-length.  (Bug #28138)
  
  ibuf_entry_build(): Instead of prefix_len, pass fixed_len to
  dtype_new_store_for_order_and_null_size().  Add debug assertions.
  
  btr_index_rec_validate(): Correct a comment about prefix indexes.
  
  rec_get_converted_size_new(), rec_convert_dtuple_to_rec_new(): Add
  debug assertions and comments.
  
  dict_col_type_assert_equal(): New debug function.
innobase/include/dict0mem.h:
  Apply innodb-5.0-* snapshots:  ss1489 and ss1547.
  
  Revision r1537:
  branches/5.0: merge r1536 and partially r1535
  
  Change the comment to a more appropriate one. Discussed with Heikki on IM.
  
  Document that DICT_MAX_INDEX_COL_LEN should not be changed.
innobase/include/os0file.h:
  Apply innodb-5.0-* snapshots:  ss1489 and ss1547.
  
  Revision r1417:
  branches/5.0: Fix Bug#9709 by retrying (forever) if ERROR_SHARING_VIOLATION or
  ERROR_LOCK_VIOLATION is encountered during file operation.
  This is caused by backup software, so InnoDB should retry while the backup
  software is done with the file.
  
  Approved by:	Heikki
innobase/include/trx0trx.h:
  Apply innodb-5.0-* snapshots:  ss1489 and ss1547.
  
  Revision r1463:
  branches/5.0: merge r1462 from trunk:
  
  Fix typo in comment.
innobase/lock/lock0lock.c:
  Apply innodb-5.0-* snapshots:  ss1489 and ss1547.
  
  Revision r1458:
  branches/5.0: Fix Bug#22819, remove assertion.  (http://bugs.mysql.com/bug.php?id=22819)
innobase/log/log0log.c:
  Apply innodb-5.0-* snapshots:  ss1489 and ss1547.
  
  Revision r1520:
  Patch to allow monitor threads to stop before proceeding with normal shutdown. 
  Also have a separate time counter for tablespace monitor.
  
  reviewed by: Heikki
  
  
  Revision r1525:
  backport of r1524
  
  Log:
  Undo bad space formatting introduced in earlier commit r1521
  
  spotted by: Marko
innobase/mtr/mtr0mtr.c:
  Apply innodb-5.0-* snapshots:  ss1489 and ss1547.
  
  Revision r1502:
  branches/5.0: Add debug code for Bug 26081. This change has some debug
  assertions that have been promoted to normal assertions. These will need
  to be undone once we've nailed this bug.
innobase/os/os0file.c:
  Apply innodb-5.0-* snapshots:  ss1489 and ss1547.
  
  Revision r1417:
  branches/5.0: Fix Bug#9709 by retrying (forever) if ERROR_SHARING_VIOLATION or
  ERROR_LOCK_VIOLATION is encountered during file operation.
  This is caused by backup software, so InnoDB should retry while the backup
  software is done with the file.
  
  Approved by:	Heikki
innobase/rem/rem0rec.c:
  Apply innodb-5.0-* snapshots:  ss1489 and ss1547.
  
  Revision r1547:
  branches/5.0: Merge r1546 from trunk:
  
  When buffering an insert to a prefix index of a variable-length column,
  do not incorrectly mark the column as fixed-length.  (Bug #28138)
  
  ibuf_entry_build(): Instead of prefix_len, pass fixed_len to
  dtype_new_store_for_order_and_null_size().  Add debug assertions.
  
  btr_index_rec_validate(): Correct a comment about prefix indexes.
  
  rec_get_converted_size_new(), rec_convert_dtuple_to_rec_new(): Add
  debug assertions and comments.
  
  dict_col_type_assert_equal(): New debug function.
innobase/row/row0sel.c:
  Apply innodb-5.0-* snapshots:  ss1489 and ss1547.
  
  Revision r1456:
  branches/5.0: merge r1452 from trunk:
  
  Fix phantom reads (http://bugs.mysql.com/27197) following Heikki's
  patch in the bug followup.
innobase/srv/srv0srv.c:
  Apply innodb-5.0-* snapshots:  ss1489 and ss1547.
  
  Revision r1460:
  branches/5.0: Merge r1459 from trunk:
  
  Fix typo in the comment.
  
  
  Revision r1520:
  Patch to allow monitor threads to stop before proceeding with normal shutdown. 
  Also have a separate time counter for tablespace monitor.
  
  reviewed by: Heikki
sql/ha_innodb.cc:
  Apply innodb-5.0-* snapshots:  ss1489 and ss1547.
  
  Revision r1436:
  branches/5.0: Fix Bug#27294 by using trx returned by check_trx_exists()
  instead of prebuilt->trx. This has been fixed in 5.1 in r782.
  
  Approved by:	Heikki
  
  
  Revision r1443:
  branches/5.0: merge r1442 from trunk:
  
  Potential fix for Bug#25645:
  
  "Move innobase_release_stat_resources(trx) outside the 'if' in
  ha_innobase::external_lock(). That would add more safety that whatever
  MySQL does at a query end, there would be no risk of a hang on the btr
  search latch."
  
  Also call innobase_release_temporary_latches() in the beginning of
  ha_innobase::close().
  
  Approved by:	Heikki
  
  
  Revision r1454:
  branches/5.0: merge r1453 from trunk:
  
  Bugfix: only call innobase_release_temporary_latches() in case of current_thd
  is not NULL, otherwise we get NULL pointer dereferencing.
  
  
  
  Revision r1504:
  branches/5.0: Apply patch for Bug 27650 from MySQL.
  
  
  Revision r1539:
  Backport of r1538 from 5.1
  Do not return error in ha_innobase::info if srv_force_recovery >= 4. This is to allow for
  normal processing of the query by MySQL instead of generating an error.
  
  Reviewed by: Heikki

innobase/btr/btr0btr.c

@@ -2542,8 +2542,11 @@ btr_index_rec_validate(
 
 		rec_get_nth_field(rec, offsets, i, &len);
 
-		/* Note that prefix indexes are not fixed size even when
-		their type is CHAR. */
+		/* Note that if fixed_size != 0, it equals the
+		length of a fixed-size column in the clustered index.
+		A prefix index of the column is of fixed, but different
+		length.  When fixed_size == 0, prefix_len is the maximum
+		length of the prefix index column. */
 
 		if ((dict_index_get_nth_field(index, i)->prefix_len == 0
 		    && len != UNIV_SQL_NULL && fixed_size

innobase/buf/buf0buf.c

@@ -1220,7 +1220,8 @@ buf_page_get_gen(
 		ut_ad(buf_validate());
 	}
 #endif
-	ut_ad(block->buf_fix_count > 0);
+	/* This debug code has been upgraded to narrow down Bug# 26081 */
+	ut_a(block->buf_fix_count > 0);
 	ut_ad(block->state == BUF_BLOCK_FILE_PAGE);
 
 	if (mode == BUF_GET_NOWAIT) {
@@ -1238,6 +1239,13 @@ buf_page_get_gen(
 		if (!success) {
 			mutex_enter(&block->mutex);
 
+			/* This debug code has been added to narrow
+			down Bug# 26081 */
+			if (UNIV_UNLIKELY(block->buf_fix_count == 0)) {
+				buf_print_diagnostic(mtr, block);
+				ut_error;
+			}
+
 			block->buf_fix_count--;
 
 			mutex_exit(&block->mutex);
@@ -1362,7 +1370,14 @@ buf_page_optimistic_get_func(
 
 	if (UNIV_UNLIKELY(!success)) {
 		mutex_enter(&block->mutex);
-		
+
+		/* This debug code has been added to narrow
+		down Bug# 26081 */
+		if (UNIV_UNLIKELY(block->buf_fix_count == 0)) {
+			buf_print_diagnostic(mtr, block);
+			ut_error;
+		}
+
 		block->buf_fix_count--;
 
 		mutex_exit(&block->mutex);
@@ -1384,7 +1399,14 @@ buf_page_optimistic_get_func(
 		}
 
 		mutex_enter(&block->mutex);
-		
+
+		/* This debug code has been added to narrow
+		down Bug# 26081 */
+		if (UNIV_UNLIKELY(block->buf_fix_count == 0)) {
+			buf_print_diagnostic(mtr, block);
+			ut_error;
+		}
+
 		block->buf_fix_count--;
 
 		mutex_exit(&block->mutex);
@@ -1404,7 +1426,10 @@ buf_page_optimistic_get_func(
 		ut_ad(buf_validate());
 	}
 #endif
-	ut_ad(block->buf_fix_count > 0);
+	/* This debug code has been upgraded from ut_ad() to narrow
+	down Bug# 26081 */
+	ut_a(block->buf_fix_count > 0);
+
 	ut_ad(block->state == BUF_BLOCK_FILE_PAGE);
 
 #ifdef UNIV_DEBUG_FILE_ACCESSES
@@ -1493,7 +1518,14 @@ buf_page_get_known_nowait(
 	
 	if (!success) {
 		mutex_enter(&block->mutex);
-		
+
+		/* This debug code has been added to narrow
+		down Bug# 26081 */
+		if (UNIV_UNLIKELY(block->buf_fix_count == 0)) {
+			buf_print_diagnostic(mtr, block);
+			ut_error;
+		}
+
 		block->buf_fix_count--;
 
 		mutex_exit(&block->mutex);
@@ -1514,7 +1546,8 @@ buf_page_get_known_nowait(
 		ut_ad(buf_validate());
 	}
 #endif
-	ut_ad(block->buf_fix_count > 0);
+	/* This debug code has been upgraded to narrow down Bug# 26081 */
+	ut_a(block->buf_fix_count > 0);
 	ut_ad(block->state == BUF_BLOCK_FILE_PAGE);
 #ifdef UNIV_DEBUG_FILE_ACCESSES
 	ut_a(block->file_page_was_freed == FALSE);
@@ -2027,7 +2060,6 @@ buf_pool_invalidate(void)
 	mutex_exit(&(buf_pool->mutex));
 }
 
-#ifdef UNIV_DEBUG
 /*************************************************************************
 Validates the buffer buf_pool data structure. */
 
@@ -2227,7 +2259,6 @@ buf_print(void)
 
 	ut_a(buf_validate());
 }	
-#endif /* UNIV_DEBUG */
 
 /*************************************************************************
 Returns the number of latched pages in the buffer pool. */
@@ -2483,3 +2514,64 @@ buf_get_free_list_len(void)
 
 	return(len);
 }
+
+/*************************************************************************
+Print the block fields. */
+
+void
+buf_block_print(
+/*============*/
+	const buf_block_t*	block)		/* in: block to print */
+{
+	fprintf(stderr,
+		"BLOCK fields\nmagic_n: 0x%lx\n"
+		"state: %lu frame: 0x%lx space:offset: 0x%lx:0x%lx\n"
+		"hash value: 0x%lx  check_index_page_at_flush: %ld\n"
+		"newest_modification %lu:%lu oldest_modification %lu:%lu\n"
+		"flush_type: %lu in_free_list: %ld in_LRU_list: %ld\n"
+		"LRU_position: %ld freed_page_clock: %lu old: %ld\n"
+		"accessed: %ld buf_fix_count: %lu io_fix: %ld "
+		"modify_clock: %lu:%lu\n"
+		"n_hash_helps: %lu n_fields: %ld n_bytes: %lu side: %lu\n"
+		"is_hashed: %lu n_pointers: %lu curr_n_fields: %lu\n"
+		"curr_n_bytes: %lu curr_side: %lu index name: %s\n"
+		"file_page_was_freed: %lu\n",
+		block->magic_n, block->state, (ulint) block->frame,
+		block->space, block->offset, block->lock_hash_val,
+		block->check_index_page_at_flush,
+		ut_dulint_get_high(block->newest_modification),
+		ut_dulint_get_low(block->newest_modification),
+		ut_dulint_get_high(block->oldest_modification),
+		ut_dulint_get_low(block->oldest_modification),
+		block->flush_type, block->in_free_list, block->in_LRU_list,
+		block->LRU_position, block->freed_page_clock,
+		block->old, block->accessed, block->buf_fix_count,
+		block->io_fix,
+		ut_dulint_get_high(block->modify_clock),
+		ut_dulint_get_low(block->modify_clock),
+		block->n_hash_helps, block->n_fields, block->n_bytes,
+		block->side, block->is_hashed, block->n_pointers,
+		block->curr_n_fields, block->curr_n_bytes,
+		block->curr_side, block->index->name,
+		block->file_page_was_freed);
+}
+
+/************************************************************************
+Print some diagnostics related to the buffer pool.*/
+
+void
+buf_print_diagnostic(
+/*=================*/
+	mtr_t*			mtr,	/* in: mtr to print */
+	const buf_block_t*	block)	/* in: block to print */
+{
+	fprintf(stderr, "=== MTR ===\n");
+	mtr_print(mtr);
+	buf_LRU_print();
+	buf_print();
+	buf_LRU_validate();
+	buf_print_io(stderr);
+	fprintf(stderr, "=== BLOCK ===\n");
+	buf_block_print(block);
+}
+

innobase/buf/buf0lru.c

@@ -945,7 +945,6 @@ buf_LRU_block_free_hashed_page(
 	buf_LRU_block_free_non_file_page(block);
 }
 
-#ifdef UNIV_DEBUG
 /**************************************************************************
 Validates the LRU list. */
 
@@ -1076,4 +1075,3 @@ buf_LRU_print(void)
 
 	mutex_exit(&(buf_pool->mutex));
 }
-#endif /* UNIV_DEBUG */

innobase/ibuf/ibuf0ibuf.c

@@ -1436,6 +1436,9 @@ ibuf_entry_build(
 		*buf2++ = 0; /* write the compact format indicator */
 	}
 	for (i = 0; i < n_fields; i++) {
+		ulint			fixed_len;
+		const dict_field_t*	ifield;
+
 		/* We add 4 below because we have the 4 extra fields at the
 		start of an ibuf record */
 
@@ -1443,11 +1446,30 @@ ibuf_entry_build(
 		entry_field = dtuple_get_nth_field(entry, i);
 		dfield_copy(field, entry_field);
 
+		ifield = dict_index_get_nth_field(index, i);
+		/* Prefix index columns of fixed-length columns are of
+		fixed length.  However, in the function call below,
+		dfield_get_type(entry_field) contains the fixed length
+		of the column in the clustered index.  Replace it with
+		the fixed length of the secondary index column. */
+		fixed_len = ifield->fixed_len;
+
+#ifdef UNIV_DEBUG
+		if (fixed_len) {
+			/* dict_index_add_col() should guarantee these */
+			ut_ad(fixed_len <= (ulint) entry_field->type.len);
+			if (ifield->prefix_len) {
+				ut_ad(ifield->prefix_len == fixed_len);
+			} else {
+				ut_ad(fixed_len
+				      == (ulint) entry_field->type.len);
+			}
+		}
+#endif /* UNIV_DEBUG */
+
 		dtype_new_store_for_order_and_null_size(
-				buf2 + i * DATA_NEW_ORDER_NULL_TYPE_BUF_SIZE,
-				dfield_get_type(entry_field),
-				dict_index_get_nth_field(index, i)
-				->prefix_len);
+			buf2 + i * DATA_NEW_ORDER_NULL_TYPE_BUF_SIZE,
+			dfield_get_type(entry_field), fixed_len);
 	}
 
 	/* Store the type info in buf2 to field 3 of tuple */

innobase/include/buf0buf.ic

@@ -11,6 +11,15 @@ Created 11/5/1995 Heikki Tuuri
 #include "buf0rea.h"
 #include "mtr0mtr.h"
 
+/*************************************************************************
+Print the mtr and block fields. */
+
+void
+buf_print_diagnostic(
+/*=================*/
+	mtr_t*			mtr,		/* in: mtr to print */
+	const buf_block_t*	block);		/* in: block to print */
+
 #ifdef UNIV_DEBUG
 extern ulint		buf_dbg_counter; /* This is used to insert validation
 					operations in execution in the
@@ -618,12 +627,9 @@ buf_page_release(
 					RW_NO_LATCH */
 	mtr_t*		mtr)		/* in: mtr */
 {
-	ulint	buf_fix_count;
-	
 	ut_ad(block);
 
 	ut_a(block->state == BUF_BLOCK_FILE_PAGE);
-	ut_a(block->buf_fix_count > 0);
 
 	if (rw_latch == RW_X_LATCH && mtr->modifications) {
 		mutex_enter(&buf_pool->mutex);
@@ -633,11 +639,16 @@ buf_page_release(
 
 	mutex_enter(&block->mutex);
 
+	/* This debug code has been added to narrow down Bug# 26081 */
+	if (UNIV_UNLIKELY(block->buf_fix_count == 0)) {
+		buf_print_diagnostic(mtr, block);
+		ut_error;
+	}
+
 #ifdef UNIV_SYNC_DEBUG
 	rw_lock_s_unlock(&(block->debug_latch));
 #endif
-	buf_fix_count = block->buf_fix_count;
-	block->buf_fix_count = buf_fix_count - 1;
+	--block->buf_fix_count;
 
 	mutex_exit(&block->mutex);
 

innobase/include/buf0lru.h

@@ -122,7 +122,6 @@ void
 buf_LRU_make_block_old(
 /*===================*/
 	buf_block_t*	block);	/* in: control block */
-#ifdef UNIV_DEBUG
 /**************************************************************************
 Validates the LRU list. */
 
@@ -135,7 +134,6 @@ Prints the LRU list. */
 void
 buf_LRU_print(void);
 /*===============*/
-#endif /* UNIV_DEBUG */
 
 #ifndef UNIV_NONINL
 #include "buf0lru.ic"

innobase/include/dict0dict.h

@@ -84,6 +84,17 @@ dtype_t*
 dict_col_get_type(
 /*==============*/
 	dict_col_t*	col);
+#ifdef UNIV_DEBUG
+/*************************************************************************
+Assert that a column and a data type match. */
+UNIV_INLINE
+ibool
+dict_col_type_assert_equal(
+/*=======================*/
+					/* out: TRUE */
+	const dict_col_t*	col,	/* in: column */
+	const dtype_t*		type);	/* in: data type */
+#endif /* UNIV_DEBUG */
 /*************************************************************************
 Gets the column number. */
 UNIV_INLINE

innobase/include/dict0dict.ic

@@ -23,6 +23,30 @@ dict_col_get_type(
 	return(&col->type);
 }
 
+#ifdef UNIV_DEBUG
+/*************************************************************************
+Assert that a column and a data type match. */
+UNIV_INLINE
+ibool
+dict_col_type_assert_equal(
+/*=======================*/
+					/* out: TRUE */
+	const dict_col_t*	col,	/* in: column */
+	const dtype_t*		type)	/* in: data type */
+{
+	ut_ad(col);
+	ut_ad(type);
+
+	ut_ad(col->type.mtype == type->mtype);
+	ut_ad(col->type.prtype == type->prtype);
+	ut_ad(col->type.len == type->len);
+	ut_ad(col->type.mbminlen == type->mbminlen);
+	ut_ad(col->type.mbmaxlen == type->mbmaxlen);
+
+	return(TRUE);
+}
+#endif /* UNIV_DEBUG */
+
 /*************************************************************************
 Gets the column number. */
 UNIV_INLINE

innobase/include/dict0mem.h

@@ -159,10 +159,13 @@ struct dict_col_struct{
 				in some of the functions below */
 };
 
-/* DICT_MAX_INDEX_COL_LEN is measured in bytes and is the max index column
-length + 1. Starting from 4.1.6, we set it to < 3 * 256, so that one can
-create a column prefix index on 255 characters of a TEXT field also in the
-UTF-8 charset. In that charset, a character may take at most 3 bytes. */
+/* DICT_MAX_INDEX_COL_LEN is measured in bytes and is the maximum
+indexed column length (or indexed prefix length). It is set to 3*256,
+so that one can create a column prefix index on 256 characters of a
+TEXT or VARCHAR column also in the UTF-8 charset. In that charset,
+a character may take at most 3 bytes.
+This constant MUST NOT BE CHANGED, or the compatibility of InnoDB data
+files would be at risk! */
 
 #define DICT_MAX_INDEX_COL_LEN		768
 

innobase/include/os0file.h

@@ -94,7 +94,8 @@ log. */
 #define	OS_FILE_PATH_ERROR		74
 #define	OS_FILE_AIO_RESOURCES_RESERVED	75	/* wait for OS aio resources
 						to become available again */
-#define	OS_FILE_ERROR_NOT_SPECIFIED	76
+#define	OS_FILE_SHARING_VIOLATION	76
+#define	OS_FILE_ERROR_NOT_SPECIFIED	77
 
 /* Types for aio operations */
 #define OS_FILE_READ	10

innobase/include/trx0trx.h

@@ -604,7 +604,7 @@ struct trx_struct{
 					NULL */
 	ibool		was_chosen_as_deadlock_victim;
 					/* when the transaction decides to wait
-					for a lock, this it sets this to FALSE;
+					for a lock, it sets this to FALSE;
 					if another transaction chooses this
 					transaction as a victim in deadlock
 					resolution, it sets this to TRUE */

innobase/lock/lock0lock.c

@@ -4438,14 +4438,11 @@ lock_table_queue_validate(
 	dict_table_t*	table)	/* in: table */
 {
 	lock_t*	lock;
-	ibool	is_waiting;
 
 #ifdef UNIV_SYNC_DEBUG
 	ut_ad(mutex_own(&kernel_mutex));
 #endif /* UNIV_SYNC_DEBUG */
 
-	is_waiting = FALSE;
-
 	lock = UT_LIST_GET_FIRST(table->locks);
 
 	while (lock) {
@@ -4455,12 +4452,9 @@ lock_table_queue_validate(
 	
 		if (!lock_get_wait(lock)) {
 
-			ut_a(!is_waiting);
-		
 			ut_a(!lock_table_other_has_incompatible(lock->trx, 0,
 						table, lock_get_mode(lock)));
 		} else {
-			is_waiting = TRUE;
 
 			ut_a(lock_table_has_to_wait_in_queue(lock));
 		}

innobase/log/log0log.c

@@ -3061,9 +3061,22 @@ logs_empty_and_mark_files_at_shutdown(void)
 
 	mutex_enter(&kernel_mutex);
 
-       /* Check that there are no longer transactions. We need this wait even
-        for the 'very fast' shutdown, because the InnoDB layer may have
-        committed or prepared transactions and we don't want to lose them. */
+	/* We need the monitor threads to stop before we proceed with a
+	normal shutdown. In case of very fast shutdown, however, we can
+	proceed without waiting for monitor threads. */
+
+	if (srv_fast_shutdown < 2
+	   && (srv_error_monitor_active
+	      || srv_lock_timeout_and_monitor_active)) {
+		
+		mutex_exit(&kernel_mutex);
+		
+		goto loop;
+	}
+
+	/* Check that there are no longer transactions. We need this wait even
+	for the 'very fast' shutdown, because the InnoDB layer may have
+	committed or prepared transactions and we don't want to lose them. */
 
 	if (trx_n_mysql_transactions > 0
 			|| UT_LIST_GET_LEN(trx_sys->trx_list) > 0) {
@@ -3184,22 +3197,8 @@ logs_empty_and_mark_files_at_shutdown(void)
 		goto loop;
 	}
 
-	/* The lock timeout thread should now have exited */
-
-	if (srv_lock_timeout_and_monitor_active) {
-
-		goto loop;
-	}
-
-	/* We now let also the InnoDB error monitor thread to exit */
-	
 	srv_shutdown_state = SRV_SHUTDOWN_LAST_PHASE;
 
-	if (srv_error_monitor_active) {
-
-		goto loop;
-	}
-
 	/* Make some checks that the server really is quiet */
 	ut_a(srv_n_threads_active[SRV_MASTER] == 0);
        ut_a(buf_all_freed());

innobase/mtr/mtr0mtr.c

@@ -326,7 +326,16 @@ mtr_print(
 	mtr_t*	mtr)	/* in: mtr */
 {
 	fprintf(stderr,
-	"Mini-transaction handle: memo size %lu bytes log size %lu bytes\n",
+		"Mini-transaction handle: memo size %lu bytes log "
+		"size %lu bytes\n"
+		"state: %lu modifications: %lu n_lock_recs %lu\n"
+		"log_mode: %lu start_lsn: %lu:%lu end_lsn: %lu:%lu\n",
 		(ulong) dyn_array_get_data_size(&(mtr->memo)),
-		(ulong) dyn_array_get_data_size(&(mtr->log)));
+		(ulong) dyn_array_get_data_size(&(mtr->log)),
+		mtr->state, (ulint) mtr->modifications,
+		mtr->n_log_recs, mtr->log_mode,
+		ut_dulint_get_high(mtr->start_lsn),
+		ut_dulint_get_low(mtr->start_lsn),
+		ut_dulint_get_high(mtr->end_lsn),
+		ut_dulint_get_low(mtr->end_lsn));
 }

innobase/os/os0file.c

@@ -243,7 +243,13 @@ os_file_get_last_error(
 			fprintf(stderr,
   "InnoDB: The error means mysqld does not have the access rights to\n"
   "InnoDB: the directory. It may also be you have created a subdirectory\n"
-  "InnoDB: of the same name as a data file.\n"); 
+  "InnoDB: of the same name as a data file.\n");
+		} else if (err == ERROR_SHARING_VIOLATION
+			   || err == ERROR_LOCK_VIOLATION) {
+			fprintf(stderr,
+  "InnoDB: The error means that another program is using InnoDB's files.\n"
+  "InnoDB: This might be a backup or antivirus software or another instance\n"
+  "InnoDB: of MySQL. Please close it to get rid of this error.\n");
 		} else {
 			fprintf(stderr,
   "InnoDB: Some operating system error numbers are described at\n"
@@ -260,6 +266,9 @@ os_file_get_last_error(
 		return(OS_FILE_DISK_FULL);
 	} else if (err == ERROR_FILE_EXISTS) {
 		return(OS_FILE_ALREADY_EXISTS);
+	} else if (err == ERROR_SHARING_VIOLATION
+		   || err == ERROR_LOCK_VIOLATION) {
+		return(OS_FILE_SHARING_VIOLATION);
 	} else {
 		return(100 + err);
 	}
@@ -369,6 +378,12 @@ os_file_handle_error_cond_exit(
 			|| err == OS_FILE_PATH_ERROR) {
 
 		return(FALSE);
+
+	} else if (err == OS_FILE_SHARING_VIOLATION) {
+
+		os_thread_sleep(10000000);  /* 10 sec */
+		return(TRUE);
+
 	} else {
 	        if (name) {
 	                fprintf(stderr, "InnoDB: File name %s\n", name);

innobase/rem/rem0rec.c

@@ -462,7 +462,7 @@ rec_get_converted_size_new(
 	case REC_STATUS_INFIMUM:
 	case REC_STATUS_SUPREMUM:
 		/* infimum or supremum record, 8 bytes */
-		return(size + 8); /* no extra data needed */
+		return(8); /* no extra data needed */
 	default:
 		ut_a(0);
 		return(ULINT_UNDEFINED);
@@ -473,23 +473,34 @@ rec_get_converted_size_new(
 		ulint	len	= dtuple_get_nth_field(dtuple, i)->len;
 		field = dict_index_get_nth_field(index, i);
 		type = dict_col_get_type(dict_field_get_col(field));
-		ut_ad(len != UNIV_SQL_NULL ||
-			!(dtype_get_prtype(type) & DATA_NOT_NULL));
+
+		ut_ad(dict_col_type_assert_equal(
+			      dict_field_get_col(field),
+			      dfield_get_type(dtuple_get_nth_field(
+						      dtuple, i))));
 
 		if (len == UNIV_SQL_NULL) {
 			/* No length is stored for NULL fields. */
+			ut_ad(!(dtype_get_prtype(type) & DATA_NOT_NULL));
 			continue;
 		}
 
 		ut_ad(len <= dtype_get_len(type)
 			|| dtype_get_mtype(type) == DATA_BLOB);
-		ut_ad(!field->fixed_len || len == field->fixed_len);
 
 		if (field->fixed_len) {
+			ut_ad(len == field->fixed_len);
+			/* dict_index_add_col() should guarantee this */
+			ut_ad(!field->prefix_len
+			      || field->fixed_len == field->prefix_len);
 		} else if (len < 128 || (dtype_get_len(type) < 256
 				&& dtype_get_mtype(type) != DATA_BLOB)) {
 			size++;
 		} else {
+			/* For variable-length columns, we look up the
+			maximum length from the column itself.  If this
+			is a prefix index column shorter than 256 bytes,
+			this will waste one byte. */
 			size += 2;
 		}
 		size += len;
@@ -885,6 +896,11 @@ rec_convert_dtuple_to_rec_new(
 		len = dfield_get_len(field);
 		fixed_len = dict_index_get_nth_field(index, i)->fixed_len;
 
+		ut_ad(dict_col_type_assert_equal(
+			      dict_field_get_col(dict_index_get_nth_field(
+							 index, i)),
+			      dfield_get_type(field)));
+
 		if (!(dtype_get_prtype(type) & DATA_NOT_NULL)) {
 			if (len == UNIV_SQL_NULL)
 				continue;

innobase/row/row0sel.c

@@ -3451,6 +3451,31 @@ cursor lock count is done correctly. See bugs #12263 and #12456!
 					pcur, 0, &mtr);
 
 		pcur->trx_if_known = trx;
+
+		rec = btr_pcur_get_rec(pcur);
+
+		if (!moves_up
+		    && !page_rec_is_supremum(rec)
+		    && set_also_gap_locks
+		    && !srv_locks_unsafe_for_binlog
+		    && prebuilt->select_lock_type != LOCK_NONE) {
+
+			/* Try to place a gap lock on the next index record
+			to prevent phantoms in ORDER BY ... DESC queries */
+
+			offsets = rec_get_offsets(page_rec_get_next(rec),
+						  index, offsets,
+						  ULINT_UNDEFINED, &heap);
+			err = sel_set_rec_lock(page_rec_get_next(rec),
+					       index, offsets,
+					       prebuilt->select_lock_type,
+					       LOCK_GAP, thr);
+
+			if (err != DB_SUCCESS) {
+
+				goto lock_wait_or_error;
+			}
+		}
 	} else {
 		if (mode == PAGE_CUR_G) {
 			btr_pcur_open_at_index_side(TRUE, index,

innobase/srv/srv0srv.c

@@ -1037,7 +1037,7 @@ srv_conc_enter_innodb(
 	if (!has_slept && !trx->has_search_latch
 	    && NULL == UT_LIST_GET_FIRST(trx->trx_locks)) {
 
-	        has_slept = TRUE; /* We let is sleep only once to avoid
+		has_slept = TRUE; /* We let it sleep only once to avoid
 				  starvation */
 
 		srv_conc_n_waiting_threads++;
@@ -1858,6 +1858,7 @@ srv_lock_timeout_and_monitor_thread(
 	double		time_elapsed;
 	time_t          current_time;
 	time_t		last_table_monitor_time;
+	time_t		last_tablespace_monitor_time;
 	time_t		last_monitor_time;
 	ibool		some_waits;
 	double		wait_time;
@@ -1870,6 +1871,7 @@ srv_lock_timeout_and_monitor_thread(
 	UT_NOT_USED(arg);
 	srv_last_monitor_time = time(NULL);
 	last_table_monitor_time = time(NULL);
+	last_tablespace_monitor_time = time(NULL);
 	last_monitor_time = time(NULL);
 loop:
 	srv_lock_timeout_and_monitor_active = TRUE;
@@ -1905,9 +1907,9 @@ srv_lock_timeout_and_monitor_thread(
 	    }
 
 	    if (srv_print_innodb_tablespace_monitor
-		&& difftime(current_time, last_table_monitor_time) > 60) {
+		&& difftime(current_time, last_tablespace_monitor_time) > 60) {
 
-		last_table_monitor_time = time(NULL);	
+		last_tablespace_monitor_time = time(NULL);	
 
 		fputs("================================================\n",
 			stderr);
@@ -2103,7 +2105,7 @@ srv_error_monitor_thread(
 
 	os_thread_sleep(2000000);
 
-	if (srv_shutdown_state < SRV_SHUTDOWN_LAST_PHASE) {
+	if (srv_shutdown_state < SRV_SHUTDOWN_CLEANUP) {
 
 		goto loop;
 	}

sql/ha_innodb.cc

@@ -15,7 +15,7 @@
 
 /* This file defines the InnoDB handler: the interface between MySQL and InnoDB
 NOTE: You can only use noninlined InnoDB functions in this file, because we
-have disables the InnoDB inlining in this file. */
+have disabled the InnoDB inlining in this file. */
 
 /* TODO list for the InnoDB handler in 5.0:
   - Remove the flag trx->active_trans and look at the InnoDB
@@ -2313,8 +2313,15 @@ ha_innobase::close(void)
 /*====================*/
 				/* out: 0 */
 {
+	THD*	thd;
+
   	DBUG_ENTER("ha_innobase::close");
 
+	thd = current_thd;  // avoid calling current_thd twice, it may be slow
+	if (thd != NULL) {
+		innobase_release_temporary_latches(thd);
+	}
+
 	row_prebuilt_free((row_prebuilt_t*) innobase_prebuilt);
 
     	my_free((gptr) upd_buff, MYF(0));
@@ -5305,7 +5312,12 @@ ha_innobase::info(
 
         if (srv_force_recovery >= SRV_FORCE_NO_IBUF_MERGE) {
 
-                DBUG_RETURN(HA_ERR_CRASHED);
+		/* We return success (0) instead of HA_ERR_CRASHED,
+		because we want MySQL to process this query and not
+		stop, like it would do if it received the error code
+		HA_ERR_CRASHED. */
+
+		DBUG_RETURN(0);
         }
 
 	/* We do not know if MySQL can call this function before calling
@@ -6150,6 +6162,12 @@ ha_innobase::external_lock(
 	trx->n_mysql_tables_in_use--;
 	prebuilt->mysql_has_locked = FALSE;
 
+	/* Release a possible FIFO ticket and search latch. Since we
+	may reserve the kernel mutex, we have to release the search
+	system latch first to obey the latching order. */
+
+	innobase_release_stat_resources(trx);
+
 	/* If the MySQL lock count drops to zero we know that the current SQL
 	statement has ended */
 
@@ -6158,12 +6176,6 @@ ha_innobase::external_lock(
 	        trx->mysql_n_tables_locked = 0;
 		prebuilt->used_in_HANDLER = FALSE;
 
-		/* Release a possible FIFO ticket and search latch. Since we
-		may reserve the kernel mutex, we have to release the search
-		system latch first to obey the latching order. */
-
-		innobase_release_stat_resources(trx);
-
 		if (!(thd->options & (OPTION_NOT_AUTOCOMMIT | OPTION_BEGIN))) {
                         if (trx->active_trans != 0) {
                                 innobase_commit(thd, TRUE);
@@ -6593,8 +6605,15 @@ ha_innobase::store_lock(
 						TL_IGNORE */
 {
 	row_prebuilt_t* prebuilt	= (row_prebuilt_t*) innobase_prebuilt;
+	trx_t*		trx;
+
+	/* Note that trx in this function is NOT necessarily prebuilt->trx
+	because we call update_thd() later, in ::external_lock()! Failure to
+	understand this caused a serious memory corruption bug in 5.1.11. */
+
+	trx = check_trx_exists(thd);
 
-	/* NOTE: MySQL  can call this function with lock 'type' TL_IGNORE!
+	/* NOTE: MySQL can call this function with lock 'type' TL_IGNORE!
 	Be careful to ignore TL_IGNORE if we are going to do something with
 	only 'real' locks! */
 
@@ -6624,7 +6643,7 @@ ha_innobase::store_lock(
 		used. */
 
 		if (srv_locks_unsafe_for_binlog &&
-		    prebuilt->trx->isolation_level != TRX_ISO_SERIALIZABLE &&
+		    trx->isolation_level != TRX_ISO_SERIALIZABLE &&
 		    (lock_type == TL_READ || lock_type == TL_READ_NO_INSERT) &&
 		    (thd->lex->sql_command == SQLCOM_INSERT_SELECT ||
 		     thd->lex->sql_command == SQLCOM_UPDATE ||