(no changes introduced by this merge)

Bug #54311: Crash on CHECK PARTITION after concurrent LOAD DATA
and adaptive_hash_index=OFF

(READ UNCOMMITTED access failure of off-page DYNAMIC or COMPRESSED columns).

Records that lack incompletely written externally stored columns may
be accessed by READ UNCOMMITTED transaction even without involving a
crash during an INSERT or UPDATE operation. I verified this as follows.

(1) added a delay after the mini-transaction for writing the clustered
index 'stub' record was committed (patch attached)
(2) started mysqld in gdb, setting breakpoints to the where the
assertions about READ UNCOMMITTED were added in the bug fix
(3) invoked ibtest3 --create-options=key_block_size=2
to create BLOBs in a COMPRESSED table
(4) invoked the following:
yes 'set transaction isolation level read uncommitted;
checksum table blobt3;select sleep(1);'|mysql -uroot test
(5) noted that one of the breakpoints was triggered
(return(NULL) in btr_rec_copy_externally_stored_field())

=== modified file 'storage/innodb_plugin/row/row0ins.c'
--- storage/innodb_plugin/row/row0ins.c	2010-06-30 08:17:25 +0000
+++ storage/innodb_plugin/row/row0ins.c	2010-06-30 08:17:25 +0000
@@ -2120,6 +2120,7 @@ function_exit:
 		rec_t*	rec;
 		ulint*	offsets;
 		mtr_start(&mtr);
+		os_thread_sleep(5000000);
 
 		btr_cur_search_to_nth_level(index, 0, entry, PAGE_CUR_LE,
 					    BTR_MODIFY_TREE, &cursor, 0,

=== modified file 'storage/innodb_plugin/row/row0upd.c'
--- storage/innodb_plugin/row/row0upd.c	2010-06-30 08:11:55 +0000
+++ storage/innodb_plugin/row/row0upd.c	2010-06-30 08:11:55 +0000
@@ -1763,6 +1763,7 @@ row_upd_clust_rec(
 		rec_offs_init(offsets_);
 
 		mtr_start(mtr);
+		os_thread_sleep(5000000);
 
 		ut_a(btr_pcur_restore_position(BTR_MODIFY_TREE, pcur, mtr));
 		rec = btr_cur_get_rec(btr_cur);

dict_table_get_format(index->table)

The REDUNDANT and COMPACT formats store a local 768-byte prefix of
each externally stored column. No row_ext cache is needed, but we
initialized one nevertheless. When the BLOB pointer was zero, we would
ignore the locally stored prefix as well. This triggered an assertion
failure in row_undo_mod_upd_exist_sec().

row_build(): Allow ext==NULL when a REDUNDANT or COMPACT table
contains externally stored columns.

row_undo_search_clust_to_pcur(), row_upd_store_row(): Invoke
row_build() with ext==NULL on REDUNDANT and COMPACT tables.

rb://382 approved by Jimmy Yang

columns

When the server crashes after a record stub has been inserted and
before all its off-page columns have been written, the record will
contain incomplete off-page columns after crash recovery. Such records
may only be accessed at the READ UNCOMMITTED isolation level or when
rolling back a recovered transaction in recv_recovery_rollback_active().
Skip these records at the READ UNCOMMITTED isolation level.

TODO: Add assertions for checking the above assumptions hold when an
incomplete BLOB is encountered.

btr_rec_copy_externally_stored_field(): Return NULL if the field is
incomplete.

row_prebuilt_t::templ_contains_blob: Clarify what "BLOB" means in this
context. Hint: MySQL BLOBs are not the same as InnoDB BLOBs.

row_sel_store_mysql_rec(): Return FALSE if not all columns could be
retrieved. Previously this function always returned TRUE.  Assert that
the record is not delete-marked.

row_sel_push_cache_row_for_mysql(): Return FALSE if not all columns
could be retrieved.

row_search_for_mysql(): Skip records containing incomplete off-page
columns. Assert that the transaction isolation level is READ
UNCOMMITTED.

rb://380 approved by Jimmy Yang

crash recovery"

rb://369 approved by Marko

and clarifies the invariant in dict_table_get_on_id().
      
In Mar 2007 Marko observed a crash during recovery, the crash resulted from
an UNDO operation on a system table. His solution was to acquire an X lock on
the data dictionary, this in hindsight was an overkill. It is unclear what
caused the crash, current hypothesis is that it was a memory corruption.
      
The X lock results in performance issues by when undoing changes due to
rollback during normal operation on regular tables.
      
Why the change is safe:
======================
The InnoDB code has changed since the original X lock change was made. In the
new code we always lock the data dictionary in X mode during startup when
UNDOing operations on the system tables (this is a given). This ensures that
the crash Marko observed cannot happen as long as all transactions that update
the system tables follow the standard rules by setting the appropriate DICT_OP
flag when writing the log records when they make the changes.
      
If transactions violate the above mentioned rule then during recovery (at
startup) the rollback code (see trx0roll.c) will not acquire the X lock
and we will see the crash again.  This will however be a different bug.

(no changed files)

ha_innobase::create(): Add the local variable row_type = form->s->row_type.
Adjust it to ROW_TYPE_COMPRESSED when ROW_FORMAT is not specified or inherited
but KEY_BLOCK_SIZE is. Observe the inherited ROW_FORMAT even when it is not
explicitly specified.

innodb_bug54679.test: New test, to test the bug and to ensure that there are
no regressions. (The only difference in the test result without the patch
applied is that the first ALTER TABLE changes ROW_FORMAT to Compact.)

> ------------------------------------------------------------
> revno: 3392.1.1
> revision-id: gshchepa@mysql.com-20100521184732-0jvpzinv0uwyvr2d
> parent: sven.sandberg@sun.com-20100520153801-yyhujm1qqa4eyfn0
> committer: Gleb Shchepa <gshchepa@mysql.com>
> branch nick: 53804-5.1
> timestamp: Fri 2010-05-21 22:47:32 +0400
> message:
>   Bug #53804: serious flaws in the alter database .. upgrade
>               data directory name command
>   
>   The check_db_name function has been modified to validate tails of
>   #mysql50#-prefixed database names for compliance with MySQL 5.0
>   database name encoding rules (the check_table_name function call
>   has been reused).

> ------------------------------------------------------------
> revno: 3386
> revision-id: sergey.glukhov@sun.com-20100518082821-yajhvbv1ghmlpu1n
> parent: aelkin@mysql.com-20100516170332-x8priwrdjwolc065
> committer: Sergey Glukhov <Sergey.Glukhov@sun.com>
> branch nick: mysql-5.1-bugteam
> timestamp: Tue 2010-05-18 13:28:21 +0500
> message:
>   Bug#48729 SELECT ... FROM INFORMATION_SCHEMA.ROUTINES causes memory to grow
>   Analysis showed that in case of accessing I_S table
>   ROUTINES we perform unnecessary allocations
>   with get_field() function for every processed row that
>   in their turn causes significant memory growth.
>   the fix is to avoid use of get_field().

> ------------------------------------------------------------
> revno: 3367 [merge]
> revision-id: joro@sun.com-20100504140328-srxf3c088j2twnq6
> parent: kristofer.pettersson@sun.com-20100503172109-f9hracq5pqsaomb1
> parent: joro@sun.com-20100503151651-nakknn8amrapmdp7
> committer: Georgi Kodinov <joro@sun.com>
> branch nick: B53371-5.1-bugteam
> timestamp: Tue 2010-05-04 17:03:28 +0300
> message:
>   Bug #53371: COM_FIELD_LIST can be abused to bypass table level grants.
>   
>   This is the 5.1 merge and extension of the fix.
>   The server was happily accepting paths in table name in all places a table
>   name is accepted (e.g. a SELECT). This allowed all users that have some 
>   privilege over some database to read all tables in all databases in all
>   mysql server instances that the server file system has access to.
>   Fixed by :
>   1. making sure no path elements are allowed in quoted table name when
>   constructing the path (note that the path symbols are still valid in table names
>   when they're properly escaped by the server).
>   2. checking the #mysql50# prefixed names the same way they're checked for
>   path elements in mysql-5.0.
> ------------------------------------------------------------
> Use --include-merges or -n0 to see merged revisions.

to 5.1 plugin codeline.

rb://378, approved by Marko

out NULL type columns, and return without creating the table.

rb://378 approved by Marko

> ------------------------------------------------------------
> revno: 3351.41.1
> revision-id: alexey.kopytov@sun.com-20100430111048-jdls6ofn4kkmpt09
> parent: sergey.glukhov@sun.com-20100329134249-03wyhzp5k92dzhcb
> committer: Alexey Kopytov <Alexey.Kopytov@Sun.com>
> branch nick: my51-bug48419
> timestamp: Fri 2010-04-30 15:10:48 +0400
> message:
>   Bug #48419: another explain crash..
>   
>   WHERE predicates containing references to empty tables in a
>   subquery were handled incorrectly by the optimizer when
>   executing EXPLAIN. As a result, the optimizer could try to
>   evaluate such predicates rather than just stop with
>   "Impossible WHERE noticed after reading const tables" as 
>   it would do in a non-subquery case. This led to valgrind 
>   errors and crashes.
>   
>   Fixed the code checking the above condition so that subqueries
>   are not excluded and hence are handled in the same way as top
>   level SELECTs.

> ------------------------------------------------------------
> revno: 1810.3987.14
> revision-id: davi.arnaut@sun.com-20100429132816-ictyul6d75itek22
> parent: ramil@mysql.com-20100429044232-f0pkyx8fnpszf142
> committer: Davi Arnaut <Davi.Arnaut@Sun.COM>
> branch nick: 50974-5.0
> timestamp: Thu 2010-04-29 10:28:16 -0300
> message:
>   Bug#50974: Server keeps receiving big (> max_allowed_packet) packets indefinitely.
>   
>   The server could be tricked to read packets indefinitely if it
>   received a packet larger than the maximum size of one packet.
>   This problem is aggravated by the fact that it can be triggered
>   before authentication.
>   
>   The solution is to no skip big packets for non-authenticated
>   sessions. If a big packet is sent before a session is authen-
>   ticated, a error is returned and the connection is closed.

> ------------------------------------------------------------
> revno: 3363 [merge]
> revision-id: davi.arnaut@sun.com-20100429231819-i3anwzrdasjmezvt
> parent: davi.arnaut@sun.com-20100401131522-895y8uzvv8ag44gs
> parent: davi.arnaut@sun.com-20100429132816-ictyul6d75itek22
> committer: Davi Arnaut <Davi.Arnaut@Sun.COM>
> branch nick: mysql-5.1-bugteam
> timestamp: Thu 2010-04-29 20:18:19 -0300
> message:
>   Manual merge.
> ------------------------------------------------------------
> Use --include-merges or -n0 to see merged revisions.

> ------------------------------------------------------------
> revno: 1810.3987.13
> revision-id: ramil@mysql.com-20100429044232-f0pkyx8fnpszf142
> parent: alexey.kopytov@sun.com-20100426200600-op06qy98llzpzgl1
> committer: Ramil Kalimullin <ramil@mysql.com>
> branch nick: b53237-5.0-bugteam
> timestamp: Thu 2010-04-29 08:42:32 +0400
> message:
>   Fix for bug #53237: mysql_list_fields/COM_FIELD_LIST stack smashing
>   
>   Problem: "COM_FIELD_LIST is an old command of the MySQL server, before there was real move to only
>   SQL. Seems that the data sent to COM_FIELD_LIST( mysql_list_fields() function) is not
>   checked for sanity. By sending long data for the table a buffer is overflown, which can
>   be used deliberately to include code that harms".
>   
>   Fix: check incoming data length.

The patch did not apply cleanly:
- Line numbers are completely off, roughly it is 2030 -> 1313
- What is called "pend" in the patch, is "arg_end" in the source.

> revno: 3351.14.56
> committer: Marko Mdkeld <marko.makela@oracle.com>
> branch nick: 5.1-innodb
> timestamp: Mon 2010-04-26 14:08:56 +0300
> message:
>   Add a test case for Bug #52745.

> ------------------------------------------------------------
> revno: 3351.14.47
> revision-id: marko.makela@oracle.com-20100421095033-0acvzxb8um8cms0a
> parent: marko.makela@oracle.com-20100421094032-ir4glqk46qvg2ywn
> committer: Marko Mäkelä <marko.makela@oracle.com>
> branch nick: 5.1-innodb
> timestamp: Wed 2010-04-21 12:50:33 +0300
> message:
>   dtuple_convert_big_rec(): Store locally any fields whose maximum length
>   is less than 256 bytes. (Bug #52745)
>   Add related comments and debug assertions to the "offsets"
>   functions in rem0rec.c.
>   Approved by Sunny Bains

> ------------------------------------------------------------
> revno: 3351.47.2
> revision-id: marko.makela@oracle.com-20100511104910-nim8kgguawpis7zo
> parent: marko.makela@oracle.com-20100511104500-c6kzd0bg5s42p8e9
> committer: Marko Mäkelä <marko.makela@oracle.com>
> branch nick: mysql-5.1-innodb2
> timestamp: Tue 2010-05-11 13:49:10 +0300
> message:
>   btr_page_split_and_insert(): Add an assertion
>   suggested by Sunny Bains when reviewing Bug #52964.

> ------------------------------------------------------------
> revno: 3351.47.1
> revision-id: marko.makela@oracle.com-20100511104500-c6kzd0bg5s42p8e9
> parent: vasil.dimov@oracle.com-20100510132852-cz457uqvj8iiy9mm
> committer: Marko Mäkelä <marko.makela@oracle.com>
> branch nick: mysql-5.1-innodb2
> timestamp: Tue 2010-05-11 13:45:00 +0300
> message:
>   Remove a stray expression. Spotted by Sunny Bains.

> ------------------------------------------------------------
> revno: 3351.14.74
> revision-id: marko.makela@oracle.com-20100504093128-44v6glupe1dsh0ug
> parent: marko.makela@oracle.com-20100503122859-k73bl51re93o0mt4
> committer: Marko Mäkelä <marko.makela@oracle.com>
> branch nick: 5.1-innodb
> timestamp: Tue 2010-05-04 12:31:28 +0300
> message:
>   btr_page_split_and_insert(): Correct the fix of Bug #52964.
>   When split_rec==NULL, choose the correct node pointer key (first_rec).

> ------------------------------------------------------------
> revno: 3351.14.50
> revision-id: marko.makela@oracle.com-20100421185359-8qaxoa2yyrpzwdd7
> parent: marko.makela@oracle.com-20100421102723-0i80uezbyu0ekj5d
> committer: Marko Mäkelä <marko.makela@oracle.com>
> branch nick: 5.1-innodb
> timestamp: Wed 2010-04-21 21:53:59 +0300
> message:
>   btr_page_split_and_insert(): Avoid an infinite loop. (Bug #52964)
>   
>   btr_page_tuple_smaller(): New function, refactored from
>   btr_page_split_and_insert().
>   
>   btr_page_get_split_rec(): Renamed from btr_page_get_sure_split_rec().
>   Note that a NULL return may mean that the tuple is to be inserted into
>   either the lower or upper page, to be determined by btr_page_tuple_smaller().
>   
>   btr_page_split_and_insert(): When btr_page_get_split_rec() returns NULL,
>   invoke btr_page_tuple_smaller() to determine which half-page the tuple
>   belongs to.
>   
>   Reviewed by Sunny Bains

> ------------------------------------------------------------
> revno: 3362
> revision-id: davi.arnaut@sun.com-20100401131522-895y8uzvv8ag44gs
> parent: ramil@mysql.com-20100429045409-r7r5lcyiruis15v7
> committer: Davi Arnaut <Davi.Arnaut@Sun.COM>
> branch nick: 50755-5.1
> timestamp: Thu 2010-04-01 10:15:22 -0300
> message:
>   Bug#50755: Crash if stored routine def contains version comments
>   
>   The problem was that a syntactically invalid trigger could cause
>   the server to crash when trying to list triggers. The crash would
>   happen due to a mishap in the backup/restore procedure that should
>   protect parser items which are not associated with the trigger. The
>   backup/restore is used to isolate the parse tree (and context) of
>   a statement from the load (and parsing) of a trigger. In this case,
>   a error during the parsing of a trigger could cause the improper
>   backup/restore sequence.
>   
>   The solution is to properly restore the original statement context
>   before the parser is exited due to syntax errors in the trigger body.