Bug #16754901 PARS_INFO_FREE NOT CALLED IN DICT_CREATE_ADD_FOREIGN_TO_DICTIONARY

Problem:

There are two situations here.  The constraint name is explicitly
given by the user and the constraint name is automatically generated
by InnoDB.  In the case of generated constraint name, it is formed by
adding table name as prefix.  The table names are stored internally in
my_charset_filename.  In the case of constraint name explicitly given
by the user, it is stored in UTF8 format itself.  So, in some
situations the constraint name is in utf8 and in some situations it is
in my_charset_filename format.  Hence this problem.

Solution:

Always store the foreign key constraint name in UTF-8 even when
automatically generated.

Bug #16754901 PARS_INFO_FREE NOT CALLED IN DICT_CREATE_ADD_FOREIGN_TO_DICTIONARY

Problem:

There was a memory leak in the function dict_create_add_foreign_to_dictionary().
The allocated pars_info_t object is not freed in the error code path.

Solution:

Allocate the pars_info_t object after the error checking.

rb#2368 in review

OPENING MISSING PARTITION

In the ha_innobase::open() call, for normal tables, there is no retry logic.
But for partitioned tables, there is a retry logic introduced as fix for:

http://bugs.mysql.com/bug.php?id=33349  
https://support.mysql.com/view.php?id=21080

The Bug#33349, does not provide sufficient information to analyze the original
problem.  The original problem reported by bug#33349 is also minor (just an
annoyance and no loss of functionality).  Most importantly, the retry logic
has been introduced without any associated test case.

So we are removing the retry logic for partitioned tables.  When the original
problem occurs, a different solution will be explored.

For the error code ER_TOO_LONG_IDENT, the identifier is expected in the my_error call.  So removing this line from here.  

FROM SHOW CREATE

Problem: The length of the internally generated foreign key name 
is not checked. 

Solution: The length of the internally generated foreign key name is
checked.  If it is greater than the allowed limit, an error message
is reported. Also, the constraint name is printed in the same manner
as the table name, using the system charset information.

rb://1969 approved by Marko.

will not export internal symbols.

buf_page_get_gen(): Do not attempt to decompress a compressed-only
page when mode == BUF_PEEK_IF_IN_POOL. This mode is only being used by
btr_search_drop_page_hash_when_freed(). There cannot be any adaptive
hash index pointing to a page that does not exist in uncompressed
format in the buffer pool.

innodb_buffer_pool_evict_update(): New function for debug builds, to handle
SET GLOBAL innodb_buffer_pool_evicted='uncompressed'
by evicting all uncompressed page frames of compressed tablespaces
from the buffer pool.

rb#1873 approved by Jimmy Yang

			    DIAGNOSTICS_AREA::SET_OK_STATUS

Use DBUG_RETURN() instead of return() if DBUG_ENTER() is
used in the function. This patch is to  fix the Windows 
pb2 failure on mysql-5.1

Approved by Marko. rb#1792

This is a followup to the fix of
Bug#14628410 ASSERTION `! IS_SET()' FAILED IN DIAGNOSTICS_AREA::SET_OK_STATUS
(satya.bodapati@oracle.com-20121213132316-5joz4phltx9yhjs7)

In innobase_mysql_tmpfile(): allocate/open the file after
the return(-1); statement.

The error code returned from Merge file/Temp file creation functions are
ignored.

Use the return codes of the row_merge_file_create() and innobase_mysql_tmpfile()
to return the error to caller if file creation fails.

Approved by Marko. rb#1618

The converted implicit lock should wait for the prior conflicting lock if found.

rb://1437 approved by Marko

btr_lift_page_up() writes wrong page number (different by -1) for upper than father page.
But in almost all of the cases, the father page should be root page, no upper
pages. It is very rare path.

In addition the leaf page should not be lifted unless the father page is root.
Because the branch pages should not become the leaf pages.

rb://1336 approved by Marko Makela.

Delete-mark change buffer records when resorting to a pessimistic
delete from the change buffer B-tree. Skip delete-marked records in
the change buffer merge and when estimating whether an operation can
be buffered. Without this fix, we could try to apply the same buffered
changes multiple times if the server was killed at the right moment.

In MySQL 5.5 and later: ibuf_get_volume_buffered_count_func(): Ignore
delete-marked (already processed) records.

ibuf_delete_rec(): Add a crash point before optimistic delete. If the
optimistic delete fails, flag the record processed before
mtr_commit().

ibuf_merge_or_delete_for_page(): Ignore delete-marked (already
processed) records.

Backport to 5.1: Rename btr_cur_del_unmark_for_ibuf() to
btr_cur_set_deleted_flag_for_ibuf() and add a parameter.

rb:1307 approved by Jimmy Yang

THOUGH IT IS NOT.

The following error message is misleading because it claims 
that the BLOB space is not counted.  

"ERROR 1118 (42000): Row size too large. The maximum row size for 
the used table type, not counting BLOBs, is 8126. You have to 
change some columns to TEXT or BLOBs"

When the ROW_FORMAT=compact or ROW_FORMAT=REDUNDANT is used,
the BLOB prefix is stored inline along with the row.  So 
the above error message is changed as follows depending on
the row format used:

For ROW_FORMAT=COMPRESSED or ROW_FORMAT=DYNAMIC, the error
message is as follows:

"ERROR 42000: Row size too large (> 8126). Changing some
columns to TEXT or BLOB may help. In current row format, 
BLOB prefix of 0 bytes is stored inline."

For ROW_FORMAT=COMPACT or ROW_FORMAT=REDUNDANT, the error
message is as follows:

"ERROR 42000: Row size too large (> 8126). Changing some
columns to TEXT or BLOB or using ROW_FORMAT=DYNAMIC or 
ROW_FORMAT=COMPRESSED may help. In current row
format, BLOB prefix of 768 bytes is stored inline."

rb://1252 approved by Marko Makela

ha_innodb::records_in_range(): Remove a debug assertion
that prohibits an open range (full table).

The patch by Jorgen Loland only removed the assertion from the
built-in InnoDB, not from the InnoDB Plugin.

Backporting the WL#5716, "Information schema table for InnoDB 
buffer pool information". Backporting revisions 2876.244.113, 
2876.244.102 from mysql-trunk.

rb://1175 approved by Jimmy Yang. 

This patch fixes various compilation warnings of the type
"error: narrowing conversion of 'x' from 'datatype1' to
'datatype2'

BY A CONCURRENT TRANSACTIO

The member function QUICK_RANGE_SELECT::init_ror_merged_scan() performs
a table handler clone. Innodb does not provide a clone operation.  
The ha_innobase::clone() is not there. The handler::clone() does not 
take care of the ha_innobase->prebuilt->select_lock_type.  Because of 
this what happens is that for one index we do a locking read, and 
for the other index we were doing a non-locking (consistent) read. 
The patch introduces ha_innobase::clone() member function.  
It is implemented similar to ha_myisam::clone().  It calls the 
base class handler::clone() and then does any additional operation 
required.  I am setting the ha_innobase->prebuilt->select_lock_type 
correctly. 

rb://1060 approved by Marko

Fixed not to check timeout during the check table.

Bug#13639204 64111: CRASH ON SELECT SUBQUERY WITH NON UNIQUE INDEX
The crash happened due to wrong calculation
of key length during creation of reference for
sort order index. The problem is that
keyuse->used_tables can have OUTER_REF_TABLE_BIT enabled
but used_tables parameter(create_ref_for_key() func) does
not have it. So key parts which have OUTER_REF_TABLE_BIT
are ommited and it could lead to incorrect key length
calculation(zero key length).

rb://976
approved by: Marko Makela

Add an assertion to ensure that string overflow is not happening.
Pointed by Coverity analysis.

      
There are two threads.  In one thread, dml operation is going on 
involving cascaded update operation.  In another thread, alter 
table add foreign key constraint is happening.  Under these 
circumstances, it is possible for the dml thread to access a 
dict_foreign_t object that has been freed by the ddl thread.  
The debug sync test case provides the sequence of operations.  
Without fix, the test case will crash the server (because of 
newly added assert).  With fix, the alter table stmt will return 
an error message.  
      
Backporting the fix from MySQL 5.5 to 5.1

rb:961
rb:947

GRACEFUL SHUTDOWN

During startup mysql picks up .frm files from the tmpdir directory and
tries to drop those tables in the storage engine.

The problem is that when tmpdir ends in / then ha_innobase::delete_table()
is passed a string like "/var/tmp//#sql123", then it wrongly normalizes it
to "/#sql123" and calls row_drop_table_for_mysql() which of course fails
to delete the table entry from the InnoDB dictionary cache.
ha_innobase::delete_table() returns an error but nevertheless mysql wipes
away the .frm file and the entry in the InnoDB dictionary cache remains
orphaned with no easy way to remove it.

The "no easy" way to remove it is to create a similar temporary table again,
copy its .frm file to tmpdir under "#sql123.frm" and restart mysqld with
tmpdir=/var/tmp (no trailing slash) - this way mysql will pick the .frm file
after restart and will try to issue drop table for "/var/tmp/#sql123"
(notice do double slash), ha_innobase::delete_table() will normalize it to
"tmp/#sql123" and row_drop_table_for_mysql() will successfully remove the
table entry from the dictionary cache.

The solution is to fix normalize_table_name_low() to normalize things like
"/var/tmp//table" correctly to "tmp/table".

This patch also adds a test function which invokes
normalize_table_name_low() with various inputs to make sure it works
correctly and a mtr test that calls this test function.

Reviewed by:	Marko (http://bur03.no.oracle.com/rb/r/929/)

ISSUES WITH COPYING PARTITIONED INNODB TABLES FROM LINUX TO WINDOWS

This problem was already fixed in mysql-trunk as part of bug #11755924.  I am 
backporting the fix to mysql-5.1.  

If we meet DB_TOO_MANY_CONCURRENT_TRXS during the execution tab_create_graph from row_create_table_for_mysql(), .ibd file for the table should be created already but was not deleted for the error handling.

rb:875 approved by Jimmy Yang

The counter handler_read_key (SSV::ha_read_key_count) is incremented 
incorrectly.

The mysql server maintains a per thread system_status_var (SSV)
object.  This object contains among other things the counter
SSV::ha_read_key_count. The purpose of this counter is to measure the
number of requests to read a row based on a key (or the number of
index lookups).

This counter was wrongly incremented in the
ha_innobase::innobase_get_index(). The fix removes
this increment statement (for both innodb and innodb_plugin).

The various callers of the innobase_get_index() was checked to
determine if anybody must increment this counter (if they first call
innobase_get_index() and then perform an index lookup).  It was found
that no caller of innobase_get_index() needs to worry about the
SSV::ha_read_key_count counter.

a.k.a. Bug#7975 deadlock without any locking, simple select and update

Bug#7975 was reintroduced when the storage engine API was made
pluggable in MySQL 5.1. Instead of looking at thd->lex directly, we
rely on handler::extra(). But, we were looking at the wrong extra()
flag, and we were ignoring the TRX_DUP_REPLACE flag in places where we
should obey it.

innodb_replace.test: Add tests for hopefully all affected statement
types, so that bug should never ever resurface. This kind of tests
should have been added when fixing Bug#7975 in MySQL 5.0.3 in the
first place.

rb:806 approved by Sunny Bains

In all callers of row_sel_convert_mysql_key_to_innobase(), assert
that the converted key is empty or nonempty when it should be.

In the ON UPDATE CASCADE clause of FOREIGN KEY constraints, the
calculated update vector was not fully initialized. This bug was
introduced in the InnoDB Plugin when implementing support for
ROW_FORMAT=DYNAMIC.

Additionally, the data type information was not initialized, but
apparently it has never been needed in this case.  Nevertheless, it is
not good programming practice to pass uninitialized values around.

calc_row_difference(): Declare the update field uninitialized in
Valgrind. Copy the data type information as well, except when the
field is SQL NULL. In the built-in InnoDB, initialize
ufield->extern_storage = FALSE (an initialization bug that had gone
unnoticed this far). The InnoDB Plugin and later have this flag to
dfield_t and have always initialized it properly.

row_ins_cascade_calc_update_vec(): Reduce the scope of some
pointers. Initialize orig_len. (This caused the bug in InnoDB Plugin
and later.)

row_ins_foreign_check_on_constraint(): Simplify a condition. Declare
the update vector uninitialized.

rb:771 approved by Jimmy Yang

hash index at shutdown

btr_search_disable(): Just drop the entire adaptive hash index,
without dropping every record separately.

buf_pool_clear_hash_index(): Renamed and simplified from
buf_pool_drop_hash_index(). Set block->index = NULL for every block in
the buffer pool. Do not release the btr_search_latch. The caller will
have to adjust other data structures.

Remove block->is_hashed. It is redundant, should be always equal to
block->index != NULL.

Remove btr_search_fully_disabled, btr_search_enabled_mutex, and
SYNC_SEARCH_SYS_CONF. We drop the AHI in one pass, without releasing
the btr_search_latch in between.

Replace void* with const rec_t* and add assertions on btr_search_latch
and btr_search_enabled to ha0ha.h, ha0ha.ic, ha0ha.c.

page_set_max_trx_id(): Ignore the adaptive hash index. I forgot to
push this in rb:750.

btr0sea.c: Always after acquiring btr_search_latch, check for
block->index==NULL or !btr_search_enabled. We can now set
block->index=NULL while only holding btr_search_latch in exclusive
mode. Always acquire btr_search_latch before reading block->index,
except in shortcuts when testing for block->index == NULL.

ha_clear(), ha_search(): Unused function, remove.

buf_page_peek_if_search_hashed(): Remove. This function may avoid
latching a page at the cost of doing a duplicate buf_pool->page_hash
lookup.

rb:775 approved by Inaam Rana

for compressed InnoDB tables

ha_innodb::info_low(): For calculating data_length or index_length,
use the compressed page size for compressed tables instead of UNIV_PAGE_SIZE.

rb:714 approved by Sunny Bains

The title of the bug is a little confusing. The actual fix is to
reintroduce random readahead inside InnoDB with a dynamic, global
switch innodb_random_read_ahead [default = off].

Approved by: Sunny Bains
rb://696

HA_INNOBASE::UPDATE_ROW, TEMPORARY TABLE, TABLE LOCK".

Attempt to update an InnoDB temporary table under LOCK TABLES
led to assertion failure in both debug and production builds
if this temporary table was explicitly locked for READ. The 
same scenario works fine for MyISAM temporary tables.

The assertion failure was caused by discrepancy between lock 
that was requested on the rows of temporary table at LOCK TABLES
time and by update operation. Since SQL-layer requested a 
read-lock at LOCK TABLES time InnoDB engine assumed that upcoming
statements which are going to be executed under LOCK TABLES will 
only read table and therefore should acquire only S-lock.
An update operation broken this assumption by requesting X-lock.

Possible approaches to fixing this problem are:

1) Skip locking of temporary tables as locking doesn't make any
   sense for connection-local objects.
2) Prohibit changing of temporary table locked by LOCK TABLES ... 
   READ.

Unfortunately both of these approaches have drawbacks which make 
them unviable for stable versions of server.

So this patch takes another approach and changes code in such way
that LOCK TABLES for a temporary table will always request write
lock. In 5.1 version of this patch switch from read lock to write
lock is done inside of InnoDBs handler methods as doing it on 
SQL-layer causes compatibility troubles with FLUSH TABLES WITH
READ LOCK.

In ha_innobase::create(), we check some things while holding an
exclusive lock on the data dictionary. Defer the locking and the
creation of transactions until after the checks have passed. The
THDVAR could hang due to a mutex wait (see Bug #11750569 - 41163:
deadlock in mysqld: LOCK_global_system_variables and LOCK_open), and
we want to avoid waiting while holding InnoDB mutexes.

innobase_index_name_is_reserved(): Replace the parameter trx_t with
THD, so that the test can be performed before starting an InnoDB
transaction. We only needed trx->mysql_thd.

ha_innobase::create(): Create transaction and lock the data dictionary
only after passing the basic tests.

create_table_def(): Move the IS_MAGIC_TABLE_AND_USER_DENIED_ACCESS
check to ha_innobase::create(). Assign to srv_lower_case_table_names
while holding dict_sys->mutex.

ha_innobase::delete_table(), ha_innobase::rename_table(),
innobase_rename_table(): Assign srv_lower_case_table_names as late as
possible. Here, the variable is not necessarily protected by
dict_sys->mutex.

ha_innobase::add_index(): Invoke innobase_index_name_is_reserved() and
innobase_check_index_keys() before allocating anything.

rb:618 approved by Jimmy Yang

causes future shutdown hang

InnoDB would hang on shutdown if any XA transactions exist in the
system in the PREPARED state. This has been masked by the fact that
MySQL would roll back any PREPARED transaction on shutdown, in the
spirit of Bug #12161 Xa recovery and client disconnection.

[mysql-test-run] do_shutdown_server: Interpret --shutdown_server 0 as
a request to kill the server immediately without initiating a
shutdown procedure.

xid_cache_insert(): Initialize XID_STATE::rm_error in order to avoid a
bogus error message on XA ROLLBACK of a recovered PREPARED transaction.

innobase_commit_by_xid(), innobase_rollback_by_xid(): Free the InnoDB
transaction object after rolling back a PREPARED transaction.

trx_get_trx_by_xid(): Only consider transactions whose
trx->is_prepared flag is set. The MySQL layer seems to prevent
attempts to roll back connected transactions that are in the PREPARED
state from another connection, but it is better to play it safe. The
is_prepared flag was introduced in the InnoDB Plugin.

trx_n_prepared: A new counter, counting the number of InnoDB
transactions in the PREPARED state.

logs_empty_and_mark_files_at_shutdown(): On shutdown, allow
trx_n_prepared transactions to exist in the system.

trx_undo_free_prepared(), trx_free_prepared(): New functions, to free
the memory objects of PREPARED transactions on shutdown. This is not
needed in the built-in InnoDB, because it would collect all allocated
memory on shutdown. The InnoDB Plugin needs this because of
innodb_use_sys_malloc.

trx_sys_close(): Invoke trx_free_prepared() on all remaining
transactions.

> ------------------------------------------------------------
> revno: 3452.13.4 [merge]
> revision-id: mmakela@bk-internal.mysql.com-20101011192851-u3bdt7erjkrgn90t
> parent: marko.makela@oracle.com-20101011081800-sby6kmb8n1mnryfq
> parent: jimmy.yang@oracle.com-20101011123613-guz1qgdktywmel1g
> committer: Marko Makela <mmakela@bk-internal.mysql.com>
> branch nick: mysql-5.1-security
> timestamp: Mon 2010-10-11 21:28:51 +0200
> message:
>   Merge Bug #57345, Bug #56982, Bug#53307 test from mysql-5.1-innodb
> ------------------------------------------------------------
> Use --include-merges or -n0 to see merged revisions.

"rows examined" estimates". This change implements "innodb_stats_method"
with options of "nulls_equal", "nulls_unequal" and "null_ignored".
      
rb://553 approved by Marko

MAX_FULL_NAME_LEN in InnoDB to address possible insufficient name buffer

Bug #59312 Approved by Sunny Bains

    
Code cleanup after changes for Bug 56628.  The general approach for 
InnoDB is to make a reference to each enum value whenever it is used in a
switch statement.  In addition, no default case should be used for switch 
statements on enum types.  This assures that if there is ever any change 
in the enum values, the switch will need to change to reflect it since a 
compiler warning will occur.  In this case, the enum row_type is declared 
in handler.h and could be changed for another storage engine.  If so, a 
warning will occur in the InnoDB build.  

Other changes;
* This patch uses 2 macros to help consolidate warning messages that
   need to occur twice in the single switch for row_format.
* Using row_format as the variable name to distinguish it from the enum
  type.
* Function declaration format correction.