"insert into.. select * from"

When inserting into a partitioned table using 'insert into
<target> select * from <src>', read_buffer_size bytes of memory
are allocated for each partition in the target table.

This resulted in large memory consumption when the number of
partitions are high.

This patch introduces a new method which tries to estimate the
buffer size required for each partition and limits the maximum
buffer size used to maximum of 10 * read_buffer_size, 
11 * read_buffer_size in case of monotonic partition functions.

sql/ha_partition.cc:
  Introduced a method ha_partition::estimate_read_buffer_size
  to estimate buffer size required for each partition. 
  Method ha_partition::start_part_bulk_insert updated
  to update the read_buffer_size before calling bulk upload
  in storage engines.
  Added thd in ha_partition::start_part_bulk_insert method signature.
sql/ha_partition.h:
  Introduced a method ha_partition::estimate_read_buffer_size.
  Added thd in ha_partition::start_part_bulk_insert method signature.

1. Fixes BUG#44030 - Error: (1500) Couldn't read the MAX(ID) autoinc value 
                     from the index (PRIMARY)

2. Disables the innodb-autoinc test for innodb plugin temporarily.
   The testcase for this bug has different result file for InnoDB plugin.
   Should add the testcase to Innodb suite with a different result file.

Detailed revision comments:

r5243 | sunny | 2009-06-04 03:17:14 +0300 (Thu, 04 Jun 2009) | 14 lines
branches/5.1: When the InnoDB and MySQL data dictionaries go out of sync, before
the bug fix we would assert on missing autoinc columns. With this fix we allow
MySQL to open the table but set the next autoinc value for the column to the
MAX value. This effectively disables the next value generation. INSERTs will
fail with a generic AUTOINC failure. However, the user should be able to
read/dump the table, set the column values explicitly, use ALTER TABLE to
set the next autoinc value and/or sync the two data dictionaries to resume
normal operations.

Fix Bug#44030 Error: (1500) Couldn't read the MAX(ID) autoinc value from the
index (PRIMARY)

rb://118

r5252 | sunny | 2009-06-04 10:16:24 +0300 (Thu, 04 Jun 2009) | 2 lines
branches/5.1: The version of the result file checked in was broken in r5243.

r5259 | vasil | 2009-06-05 10:29:16 +0300 (Fri, 05 Jun 2009) | 7 lines
branches/5.1:

Remove the word "Error" from the printout because the mysqltest suite
interprets it as an error and thus the innodb-autoinc test fails.

Approved by:	Sunny (via IM)
r5466 | vasil | 2009-07-02 10:46:45 +0300 (Thu, 02 Jul 2009) | 6 lines
branches/5.1:

Adjust the failing innodb-autoinc test to conform to the latest behavior
of the MySQL code. The idea and the comment in innodb-autoinc.test come
from Sunny.

      
The test case rpl_do_grant fails sporadically on PB2 with "Access
denied for user 'create_rout_db'@'localhost' ...". Inspecting the
test case, one may find that if issues a GRANT on the master
connection and immediately after it creates two new connections
(one to the master and one to the slave) using the credentials
set with the GRANT.
      
Unfortunately, there is no synchronization between master and
slave after the grant and before the connections are
established. This can result in slave not having executed the
GRANT by the time the connection is attempted.
      
This patch fixes this by deploying a sync_slave_with_master
between the grant and the connections attempt.

      
The test case creates two temporary tables, then closes the
connection, waits for it to disconnect, then syncs the slave with
the master, checks for remaining opened temporary tables on
slave (which should be 0) and finally drops the used
database (mysqltest).
      
Unfortunately, sometimes, the test fails with one open table on
the slave. This seems to be caused by the fact that waiting for
the connection to be closed is not sufficient. The test needs to
wait for the DROP event to be logged and only then synchronize
the slave with the master and proceed with the check. This is
caused by the asynchronous nature of the disconnect wrt
binlogging of the DROP temporary table statement.
      
We fix this by deploying a call to wait_for_binlog_event.inc
on the test case, which makes execution to wait for the DROP
temp tables event before synchronizing master and slave.

Problem: LOGGER::general_log_write() relied on valid "thd" parameter passed
but had inconsistent "if (thd)" check.

Fix: as we always pass a valid "thd" parameter to the method, 
redundant check removed.


sql/log.cc:
  Fix for bug#47130: misplaced or redundant check for null pointer?
    - code clean-up, as we rely on the "thd" parameter in the
  LOGGER::general_log_write(), redundant "if (thd)" check removed, 
  added assert(thd) instead.

The problem is that argument buffer can be used as result buffer
and it leads to argument value change.
The fix is to use 'old buffer' as result buffer only
if first argument is not constant item.


mysql-test/r/func_str.result:
  test result
mysql-test/t/func_str.test:
  test case
sql/item_strfunc.cc:
  The problem is that argument buffer can be used as result buffer
  and it leads to argument value change.
  The fix is to use 'old buffer' as result buffer only
  if first argument is not constant item.

In RBR, There is an inconsistency between slaves and master.
When INSERT statement which includes an auto_increment field is executed,
Store engine of master will check the value of the auto_increment field. 
It will generate a sequence number and then replace the value, if its value is NULL or empty.
if the field's value is 0, the store engine will do like encountering the NULL values 
unless NO_AUTO_VALUE_ON_ZERO is set into SQL_MODE.
In contrast, if the field's value is 0, Store engine of slave always generates a new sequence number 
whether or not NO_AUTO_VALUE_ON_ZERO is set into SQL_MODE.

SQL MODE of slave sql thread is always consistency with master's.
Another variable is related to this bug.
If generateing a sequence number is decided by the values of
table->auto_increment_field_not_null and SQL_MODE(if includes MODE_NO_AUTO_VALUE_ON_ZERO)
The table->auto_increment_is_not_null is FALSE, which causes this bug to appear. ..

partial backport of bug43138 fix


mysql-test/r/warnings.result:
  test result
mysql-test/t/warnings.test:
  test case
sql/sql_class.cc:
  partial backport of bug43138 fix
sql/sql_class.h:
  partial backport of bug43138 fix
sql/sql_table.cc:
  partial backport of bug43138 fix

on Windows in dbug.c) -- part 2: a patch for the DBUG subsystem
to detect misuse of DBUG_ENTER / DBUG_RETURN macros.
5.1 version.

            can crash under load

Merge from 5.0, after backport from 5.1/5.4 to 5.0.
This makes the fixes for
Bug 44068 (RESTORE can disable the MyISAM Key Cache)
Bug 40944 (Backup: crash after myisampack)
available to 5.1.

Archive engine returns wrong values for average record length
and max data length.

With this fix they're calculated as following:
- max data length is 2 ^ 63 where large files are supported
  and INT_MAX32 where this is not supported;
- average record length is data length / records in data file.

mysql-test/r/archive.result:
  A test case for BUG#29203.
mysql-test/t/archive.test:
  A test case for BUG#29203.
storage/archive/ha_archive.cc:
  Better estimation for average row length and maximal data
  file length.

Create temporary InnoDB table fails on case insensitive
filesystems, when lower_case_table_names is 2 (e.g. OS X)
and temporary directory path contains upper case letters.

The problem was that tmpdir prefix was converted to lower
case when table was created, but was passed as is when
table was opened.

Fixed by leaving tmpdir prefix part intact.

mysql-test/r/lowercase_mixed_tmpdir_innodb.result:
  A test case for BUG#45638.
mysql-test/t/lowercase_mixed_tmpdir_innodb-master.opt:
  A test case for BUG#45638.
mysql-test/t/lowercase_mixed_tmpdir_innodb-master.sh:
  A test case for BUG#45638.
mysql-test/t/lowercase_mixed_tmpdir_innodb.test:
  A test case for BUG#45638.
sql/handler.cc:
  Fixed get_canonical_filename() to not lowercase filesystem
  path prefix for temporary tables.

Updates the results of all the out-dated test suites and adds 
the special mysqltest command to enable innodb for the tests that need it.

            can crash under load

Backport from 5.1.
Does also include key cache fixes from:
Bug 44068 (RESTORE can disable the MyISAM Key Cache)
Bug 40944 (Backup: crash after myisampack)



include/keycache.h:
  Bug#17332 - changing key_buffer_size on a running server
              can crash under load
  Added KEY_CACHE components in_resize and waiting_for_resize_cnt.
myisam/mi_preload.c:
  Bug#17332 - changing key_buffer_size on a running server
              can crash under load
  Added code to allow LOAD INDEX to load indexes of different block size.
mysys/mf_keycache.c:
  Bug#17332 - changing key_buffer_size on a running server
              can crash under load
  .
  Changed resize_key_cache() to not disable the key cache
  after the flush phase. Changed queue handling to use
  standard functions. Wake all threads waiting on resize_queue.
  We can now have read/write threads waiting there (see below).
  .
  Combined add_to_queue() and the wait loops that were always
  following it to the new function wait_on_queue().
  Combined release_queue() and the condition that was always
  preceding it to the new function release_whole_queue().
  .
  Added code to flag and respect the exceptional situation
  BLOCK_IN_EVICTION.
  .
  Rewrote the resize branch of find_key_block().
  .
  Added code to the eviction handling in find_key_block()
  to catch more exceptional cases.
  .
  Changed key_cache_read(), key_cache_insert() and key_cache_write()
  so that they lock keycache->cache_lock whenever the key cache is
  initialized. Checking for a disabled cache and incrementing and
  decrementing the "resize counter" is always done within the lock.
  Locking and unlocking as well as counting the "resize counter" is
  now done once outside the loop. All three functions can now handle
  a NULL return from find_key_block. This happens in the flush phase
  of a resize and demands direct file I/O. Care is taken for
  secondary requests (PAGE_WAIT_TO_BE_READ) to wait in any case.
  Moved block status changes behind the copying of buffer data.
  key_cache_insert() does now read the block if the caller did
  supply less data than a full cache block.
  key_cache_write() does now take care of parallel running flushes
  (BLOCK_FOR_UPDATE, BLOCK_IN_FLUSHWRITE).
  .
  Changed free_block() to un-initialize block variables in the
  correct order and respect an exceptional BLOCK_IN_EVICTION state.
  .
  Changed flushing to take care for parallel running writes.
  Changed flushing to avoid freeing blocks in eviction.
  Changed flushing to consider that parallel writes can move blocks
  from the file_blocks hash to the changed_blocks hash.
  Changed flushing to take care for other parallel flushes.
  Changed flushing to assure that it ends with everything flushed.
  Optimized normal flush at end of statement (FLUSH_KEEP),
  but let other flush types be stringent.
  .
  Added some comments and debugging statements.
mysys/my_static.c:
  Bug#17332 - changing key_buffer_size on a running server
              can crash under load
  Removed an unused global variable.
sql/ha_myisam.cc:
  Bug#17332 - changing key_buffer_size on a running server
              can crash under load
  Moved an automatic (stack) variable to the scope where it is used.
sql/sql_table.cc:
  Bug#17332 - changing key_buffer_size on a running server
              can crash under load
  Changed TL_READ to TL_READ_NO_INSERT in mysql_preload_keys.

      
The parser rule for expressions in a udf parameter list contains 
two hacks: 
First, the parser input stream is read verbatim, bypassing 
the lexer.
Second, the Item::name field is overwritten. If the argument to a
udf was a field, the field's name as seen by name resolution was
overwritten this way.
If the field name was quoted or escaped, it would appear as e.g. "`field`".
Fixed by not overwriting field names.

mysql-test/r/udf.result:
  Bug#46259: Test result.
mysql-test/t/udf.test:
  Bug#46259: Test case.
sql/sql_yacc.yy:
  Bug#46259: Fix.

This test case uses mysqlbinlog to dump the content of master-bin.000001,
but the content of master-bin.000001 is not that this test needs.

MTR runs a lot of test cases on one server, so when this test starts, the current binlog file
might not be master-bin.000001, or there are other events are written by tests before.
'RESET MASTER' command must be called at the begin, it ensures that binlog of this test
is wrote to master-bin.000001 correctly.  

Three other tests have the same problem, They were fixed together.
mysqlbinlog-cp932
binlog_incident
binlog_tmp_table

Postfix.
extra/rpl_tests/rpl_row_sp006.test had changed to fix this bug.
extra/rpl_tests/rpl_row_sp006.test is also referenced by rpl_ndb_sp006,
So rpl_row_sp006.result must be changed too.

 
The external 'for' loop in remove_dup_with_compare() handled 
HA_ERR_RECORD_DELETED by just starting over without advancing 
to the next record which caused an infinite loop. 
 
This condition could be triggered on certain data by a SELECT 
query containing DISTINCT, GROUP BY and HAVING clauses. 

Fixed remove_dup_with_compare() so that we always advance to 
the next record when receiving HA_ERR_RECORD_DELETED from 
rnd_next(). 

mysql-test/r/distinct.result:
  Added a test case for bug #46159.
mysql-test/t/distinct.test:
  Added a test case for bug #46159.
sql/sql_select.cc:
  Fixed remove_dup_with_compare() so that we always advance to 
  the next record when receiving HA_ERR_RECORD_DELETED from 
  rnd_next().

           The plugin feature is disabled, you need HAVE_DLOPEN

Selectively skip tests that require dynamic loading (ie: static builds).

mysql-test/include/have_dynamic_loading.inc:
  Add require file.
mysql-test/t/ps_not_windows.test:
  Test requires dynamic loading support.

(Backport)

Problem is that when insert (ha_start_bulk_insert) in i partitioned table,
it will call ha_start_bulk_insert for every partition, used or not.

Solution is to delay the call to the partitions ha_start_bulk_insert until
the first row is to be inserted into that partition


sql/ha_partition.cc:
  Bug#35845: unneccesary call to ha_start_bulk_insert for not used partitions
  
  Using a bitmap for keeping record of which partitions for which
  ha_start_bulk_insert has been called, and check against that if one
  should call it before continue with the insert/update, or if it has already
  been called.
  
  This way it will only call ha_start_bulk_insert for the used partitions.
  There is also a little prediction on how many rows that will be inserted into
  the current partition, it will guess on equal distribution of the records
  across all partitions, accept for the first used partition, which will guess
  at 50% of the given estimate, if it is a monotonic partitioning function.
sql/ha_partition.h:
  Bug#35845: unneccesary call to ha_start_bulk_insert for not used partitions
  
  Added help variables and function for delaying ha_bulk_insert until it has
  to be called.
  
  Fixed a comment.

Incremental patch part 2

Removing dead code and changing a note level message to a warning.


sql/sql_plugin.cc:
  * Remove free_slots. The only purpose for this variable was to trigger
    a redundant warning message  and it failed.
  * Change the note level message about shutting down plugins which
    didn't end nicely to a warning level message. (If this shutdown
    fails and there still are reference counts in the plugin an
    additional error level message is emitted)

on subquery inside a SP 

Problem: repeated call of a SP containing an incorrect query with a 
subselect may lead to failed ASSERT().

Fix: set proper sublelect's state in case of error occured during 
subquery transformation.


mysql-test/r/sp.result:
  Fix for bug#46629: Item_in_subselect::val_int(): Assertion `0' 
  on subquery inside a SP 
    - test result.
mysql-test/t/sp.test:
  Fix for bug#46629: Item_in_subselect::val_int(): Assertion `0' 
  on subquery inside a SP 
    - test case.
sql/item_subselect.cc:
  Fix for bug#46629: Item_in_subselect::val_int(): Assertion `0' 
  on subquery inside a SP 
    - don't set Item_subselect::changed in the Item_subselect::fix_fields()
  if an error occured during subquery transformation.
  That prevents us of further processing incorrect subqueries after 
  Item_in_subselect::select_in_like_transformer().