can crash under load

Resizing a key cache while it was in heavy use could crash the
server. There were several race conditions.

I reworked some of the algorithms to fix the race conditions.

No test case. Repeating the crashes requires heavy concurrent
load on the key cache. A test script is attached to the bug report.

More explanations to the changes are contained in a text file
attached to the bug report.


include/keycache.h:
  Bug#17332 - changing key_buffer_size on a running server
              can crash under load
  Added KEY_CACHE components in_resize and waiting_for_resize_cnt.
mysys/mf_keycache.c:
  Bug#17332 - changing key_buffer_size on a running server
              can crash under load
  
  Changed resize_key_cache() to not disable the key cache
  after the flush phase. Changed queue handling to use
  standard functions. Wake all threads waiting on resize_queue.
  We can now have read/write threads waiting there (see below).
  
  Combined add_to_queue() and the wait loops that were always
  following it to the new function wait_on_queue().
  Combined release_queue() and the condition that was always
  preceding it to the new function release_whole_queue().
  
  Added code to flag and respect the exceptional situation
  BLOCK_IN_EVICTION.
  
  Rewrote the resize branch of find_key_block().
  
  Added code to the eviction handling in find_key_block()
  to catch more exceptional cases.
  
  Changed key_cache_read(), key_cache_insert() and key_cache_write()
  so that they lock keycache->cache_lock whenever the key cache is
  initialized. Checking for a disabled cache and incrementing and
  decrementing the "resize counter" is always done within the lock.
  Locking and unlocking as well as counting the "resize counter" is
  now done once outside the loop. All three functions can now handle
  a NULL return from find_key_block. This happens in the flush phase
  of a resize and demands direct file I/O. Care is taken for
  secondary requests (PAGE_WAIT_TO_BE_READ) to wait in any case.
  Moved block status changes behind the copying of buffer data.
  key_cache_insert() does now read the block if the caller did
  supply less data than a full cache block.
  key_cache_write() does now take care of parallel running flushes
  (BLOCK_FOR_UPDATE, BLOCK_IN_FLUSHWRITE).
  
  Changed free_block() to un-initialize block variables in the
  correct order and respect an exceptional BLOCK_IN_EVICTION state.
  
  Changed flushing to take care for parallel running writes.
  Changed flushing to avoid freeing blocks in eviction.
  Changed flushing to consider that parallel writes can move blocks
  from the file_blocks hash to the changed_blocks hash.
  Changed flushing to take care for other parallel flushes.
  Changed flushing to assure that it ends with everything flushed.
  
  Added some comments and debugging statements. ;-)
mysys/my_static.c:
  Bug#17332 - changing key_buffer_size on a running server
              can crash under load
  Removed an unused global variable.
sql/handler.cc:
  Bug#17332 - changing key_buffer_size on a running server
              can crash under load
  Changed types of local variables to match their use
  for init_key_cache().
sql/sql_table.cc:
  Bug#17332 - changing key_buffer_size on a running server
              can crash under load
  Changed TL_READ to TL_READ_NO_INSERT in mysql_preload_keys.
storage/myisam/ha_myisam.cc:
  Bug#17332 - changing key_buffer_size on a running server
              can crash under load
  Moved an automatic (stack) variable to the scope where it is used.
storage/myisam/mi_preload.c:
  Bug#17332 - changing key_buffer_size on a running server
              can crash under load
  Added some preliminary code to
  - allow LOAD INDEX to load indexes of different block size,
  - align load chunks to key cache blocks.

into  mysql.com:/home/svoj/devel/mysql/merge/mysql-5.1-engines


mysql-test/r/symlink.result:
  Auto merged
mysql-test/t/symlink.test:
  Auto merged
sql/ha_ndbcluster.cc:
  Auto merged
sql/mysqld.cc:
  Auto merged
sql/sql_class.cc:
  Auto merged
sql/table.cc:
  Auto merged
mysql-test/r/myisam.result:
  Use local.
mysql-test/t/myisam.test:
  Use local.

into  mysql.com:/home/svoj/devel/mysql/merge/mysql-5.1-engines


mysql-test/r/symlink.result:
  Auto merged
mysql-test/t/symlink.test:
  Auto merged
sql/lock.cc:
  Auto merged
sql/opt_range.cc:
  Auto merged
sql/sql_class.h:
  Auto merged
sql/sql_insert.cc:
  Auto merged
sql/sql_parse.cc:
  Auto merged
sql/table.cc:
  Auto merged
storage/federated/ha_federated.cc:
  Auto merged
storage/myisam/mi_check.c:
  Auto merged
storage/myisam/mi_packrec.c:
  Auto merged
mysql-test/r/myisam.result:
  Manually merged.
mysql-test/t/myisam.test:
  Manually merged.

into  mysql.com:/home/svoj/devel/mysql/merge/mysql-5.0-engines


mysql-test/t/myisam.test:
  Auto merged

into  mysql.com:/home/svoj/devel/mysql/merge/mysql-5.0-engines


mysql-test/r/symlink.result:
  Auto merged
mysql-test/t/symlink.test:
  Auto merged
sql/table.cc:
  Auto merged
mysql-test/r/myisam.result:
  Manually merged.
mysql-test/t/myisam.test:
  Manually merged.

into  chilla.local:/home/mydev/mysql-5.0-bug24607


mysql-test/r/myisam.result:
  Auto merged
mysql-test/t/myisam.test:
  Auto merged

into  chilla.local:/home/mydev/mysql-5.1-bug24607


mysql-test/r/myisam.result:
  Auto merged
mysql-test/t/myisam.test:
  Auto merged

into  chilla.local:/home/mydev/mysql-5.1-bug24607


mysql-test/t/myisam.test:
  Auto merged
mysql-test/r/myisam.result:
  Manual merged

After merge fix

into  chilla.local:/home/mydev/mysql-5.0-bug24607


mysql-test/r/myisam.result:
  Manual merged
mysql-test/t/myisam.test:
  Manual merged

Fixed test. On 32-bit machines which compile without
-DBIG_TABLES, MAX_ROWS is truncated to a 32-bit value.
Using a value below 4G is portable.


mysql-test/r/myisam.result:
  Bug#24607 - MyISAM pointer size determined incorrectly
  Fixed test results.

into  kahlann.erinye.com:/home/df/mysql/build/mysql-5.0-build-work


sql/ha_ndbcluster.cc:
  Auto merged

into  kahlann.erinye.com:/home/df/mysql/build/mysql-5.1-build-work

into  mysql.com:/home/svoj/devel/mysql/BUG24401/mysql-5.1-engines


mysql-test/r/myisam.result:
  Auto merged
mysql-test/t/myisam.test:
  Auto merged

into  mysql.com:/home/svoj/devel/mysql/BUG24401/mysql-5.1-engines

into  mysql.com:/home/svoj/devel/mysql/BUG24401/mysql-5.0-engines


mysql-test/r/myisam.result:
  Auto merged
mysql-test/t/myisam.test:
  Auto merged

into  mysql.com:/home/svoj/devel/mysql/BUG24401/mysql-5.0-engines

into  mysql.com:/home/svoj/devel/mysql/BUG24401/mysql-4.1-engines

into  willster.(none):/home/stewart/Documents/MySQL/5.0/ndb-work

into  willster.(none):/home/stewart/Documents/MySQL/5.1/ndb-work


storage/ndb/src/common/util/SocketClient.cpp:
  Auto merged

ndb/src/common/util/SocketClient.cpp:
  fix two problems recently introduced:
  - HPUX build problem
  - some connect errors not being detected properly

into  poseidon.mysql.com:/home/tomas/mysql-5.0-ndb


sql/mysqld.cc:
  Auto merged
sql/sql_class.cc:
  Auto merged

into  chilla.local:/home/mydev/mysql-4.1-bug24607


mysql-test/r/myisam.result:
  Auto merged
mysql-test/t/myisam.test:
  Auto merged

into  production.mysql.com:/usersnfs/mjorgensen/bktrees/mysql-5.1-build


sql/ha_ndbcluster.cc:
  Auto merged
sql/mysqld.cc:
  Auto merged

into  chilla.local:/home/mydev/mysql-5.0-bug24607


mysql-test/r/myisam.result:
  Auto merged
mysql-test/t/myisam.test:
  Auto merged

into  xiphis.org:/home/antony/work2/p1-bug25396.5


sql/ha_ndbcluster.cc:
  Auto merged
sql/mysqld.cc:
  Auto merged

into  chilla.local:/home/mydev/mysql-5.1-bug24607

into  clam.ndb.mysql.com:/export/space/pekka/ndb/version/my50-ndb

into  clam.ndb.mysql.com:/export/space/pekka/ndb/version/my51-ndb

into  mysql.com:/data0/mysqldev/users/tomas/mysql-5.1-build


configure.in:
  Auto merged
cmd-line-utils/readline/undo.c:
  Auto merged
sql/ha_ndbcluster.cc:
  Auto merged
storage/myisam/mi_packrec.c:
  Auto merged

into  willster.(none):/home/stewart/Documents/MySQL/5.0/bug25487

into  willster.(none):/home/stewart/Documents/MySQL/5.1/bug25567


mysql-test/r/ndb_basic.result:
  Auto merged

into  willster.(none):/home/stewart/Documents/MySQL/5.1/bug25567


storage/ndb/include/mgmapi/mgmapi.h:
  Auto merged
storage/ndb/include/util/SocketClient.hpp:
  Auto merged
storage/ndb/src/common/transporter/Transporter.cpp:
  Auto merged
storage/ndb/src/common/util/SocketClient.cpp:
  Auto merged
storage/ndb/src/mgmapi/mgmapi.cpp:
  Auto merged

mysql-test/r/ndb_basic.result:
  from spaces, to tabs. what fun!
sql/ha_ndbcluster.h:
  fix estimate_rows_upper_bound() to be correct for NDB
sql/sql_select.cc:
  use non-exact records information

  "Valgrind leak in closecon_handlerton"
  plugin_shutdown() calls plugin_deinitialize() which calls ha_finalize_handlerton().
  ndbcluster_end() fails to wait for the ndb utility thread to exit which results in
  the handlerton struct being freed before the ndb utility thread has destroyed it's THD
  but before the plugin has been marked as UNINITIALIZED

Bug is caused by misuse of abort_loops variable and not locking mutex during calls to
pthread condition variable functions causing a race in valgrind's pthread_cond_wait
implementation.


sql/ha_ndbcluster.cc:
  Bug25396
  Valgrind requires that mutex be held during call to pthread_cond_signal.
  Change pthread_cond_timedwait() to pthread_cond_wait() where the timeout is not needed.
  Ensure that appropiate variables are protected by mutex.
  Remove use of abort_loop global variable.
  Ensure that ndbcluster_end waits for util thread to exit.
  Add an extra cond_var as insurance against non-conforming pthreads implementations.
sql/mysqld.cc:
  Bug25386
  Valgrind requires that mutex be held during call to pthread_cond_signal.
BUILD/compile-amd64-valgrind-max:
  New BitKeeper file ``BUILD/compile-amd64-valgrind-max''

into  mysql.com:/home/svoj/devel/mysql/BUG24401/mysql-5.1-engines


storage/myisam/mi_open.c:
  Auto merged

into  mysql.com:/home/svoj/devel/mysql/BUG24401/mysql-5.0-engines


myisam/mi_open.c:
  Auto merged

into  clam.ndb.mysql.com:/export/space/pekka/ndb/version/my51-ndb


sql/ha_ndbcluster.cc:
  Auto merged