WL#3397 Refactoring storage engine test cases (for falcon)
    It contains also fixes according to code review.
    Contents: Testcases which were in history dedicated to InnoDB or MyISAM only.
    Modifications:
    1. Shift the main testing code into include/<testing field>.inc
       Introduce $variables which can be used to omit tests for features which are not supported by
       certain storage engines.
    2. The storage engine to be tested is assigned within the toplevel script (t/<whatever>_<engine>.test)
       via variable $engine_type and the the main testing code is sourced from 
       include/<testing field>.inc
    3. Some toplevel testscripts have to be renamed to
       - avoid immediate or future namespace clashes
       - show via filename which storage engine is tested
    4. Minor code cleanup like remove trailing spaces, some additional comments ....

 Fixing test result.

The Item::tmp_table_field_from_field_type() function creates Field_datetime
object instead of Field_timestamp object for timestamp field thus always
changing data type is a tmp table is used.

The Field_blob object constructor which is used in the 
Item::tmp_table_field_from_field_type() is always setting packlength field of
newly created blob to 4. This leads to changing fields data type for example
from the blob to the longblob if a temporary table is used.

The Item::make_string_field() function always converts Field_string objects 
to Field_varstring objects. This leads to changing data type from the 
char/binary to varchar/varbinary.

Added appropriate Field_timestamp object constructor for using in the 
Item::tmp_table_field_from_field_type() function.

Added Field_blob object constructor which sets pack length according to
max_length argument.

The Item::tmp_table_field_from_field_type() function now creates
Field_timestamp object for a timestamp field.

The Item_type_holder::display_length() now returns correct NULL length NULL
length. 

The Item::make_string_field() function now doesn't change Field_string to
Field_varstring in the case of Item_type_holder. 

The Item::tmp_table_field_from_field_type() function now uses the Field_blob
constructor which sets packlength according to max_length.

 Fixed BUG#19542 "InnoDB doesn't increase the Handler_read_prev couter".
 Fixed BUG#19609 "Case sensitivity of innodb_data_file_path gives stupid error".
 Fixed BUG#19727 "InnoDB crashed server and crashed tables are ot recoverable".
 Also:
 * Remove remnants of the obsolete concept of memoryfixing tables and indexes.
 * Remove unused dict_table_LRU_trim().
 * Remove unused 'trx' parameter from dict_table_get_on_id_low(),
   dict_table_get(), dict_table_get_and_increment_handle_count().
 * Add a normal linked list implementation.
 * Add a work queue implementation.
 * Add 'level' parameter to mutex_create() and rw_lock_create().
   Remove mutex_set_level() and rw_lock_set_level().
 * Rename SYNC_LEVEL_NONE to SYNC_LEVEL_VARYING.
 * Add support for bound ids in InnoDB's parser.
 * Define UNIV_BTR_DEBUG for enabling consistency checks of
   FIL_PAGE_NEXT and FIL_PAGE_PREV when accessing sibling
   pages of B-tree indexes.
   btr_validate_level(): Check the validity of the doubly linked
   list formed by FIL_PAGE_NEXT and FIL_PAGE_PREV.
 * Adapt InnoDB to the new tablename to filename encoding in MySQL 5.1.
   ut_print_name(), ut_print_name1(): Add parameter 'table_id' for
   distinguishing names of tables from other identifiers.
   New: innobase_convert_from_table_id(), innobase_convert_from_id(),
        innobase_convert_from_filename(), innobase_get_charset.
   dict_accept(), dict_scan_id(), dict_scan_col(), dict_scan_table_name(),
   dict_skip_word(), dict_create_foreign_constraints_low(): Add
   parameter 'cs' so that isspace() can be replaced with my_isspace(),
   whose operation depends on the connection character set.
   dict_scan_id(): Convert identifier to UTF-8.
   dict_str_starts_with_keyword(): New extern function, to replace
   dict_accept() in row_search_for_mysql().
   mysql_get_identifier_quote_char(): Replaced with innobase_print_identifier().
   ha_innobase::create(): Remove the thd->convert_strin() call. Pass the
   statement to InnoDB in the connection character set and let InnoDB
   convert the identifier to UTF-8.
 * Add max_row_size to dict_table_t.
 * btr0cur.c
   btr_copy_externally_stored_field(): Only set the 'offset' variable
   when needed.
 * buf0buf.c
   buf_page_io_complete(): Write to the error log if the page number or
   the space id o the disk do not match those in memory. Also write to
   the error log if a page was read from the doublewrite buffer. The
   doublewrite buffer should be only read by the lower-level function
   fil_io() at database startup.
 * dict0dict.c
   dict_scan_table_name(): Remove fallback to differently encoded name
   when the table is not found. The encoding is handled at a higher level.
 * ha_innodb.cc
   Increment statistic counter in ha_innobase::index_prev() (bug 19542).
   Add innobase_convert_string wrapper function and a new file
   ha_prototypes.h.
   innobase_print_identifier(): Remove TODO comment before calling
   get_quote_char_for_identifier(). That function apparently assumes
   the identifier to be encoded in UTF-8.
 * ibuf0ibuf.c|h
   ibuf_count_get(), ibuf_counts[], ibuf_count_inited(): Define these
   only #ifdef UNIV_IBUF_DEBUG. Previously, when compiled without
   UNIV_IBUF_DEBUG, invoking ibuf_count_get() would crash InnoDB.
   The function is only being called #ifdef UNIV_IBUF_DEBUG.
 * innodb.result
   Adjust the results for changes in the foreign key error messages.
 * mem0mem.c|h
   New: mem_heap_dup(), mem_heap_printf(), mem_heap_cat().
 * os0file.c
   Check the page trailers also after writing to disk. This improves
   chances of diagnosing bug 18886.
   os_file_check_page_trailers(): New function for checking that the
   two copies of the LSN stamped on the page match.
   os_aio_simulated_handle(): Call os_file_check_page_trailers()
   before and after os_file_write().
 * row0mysql.c
   Move trx_commit_for_mysql(trx) calls before calls to
   row_mysql_unlock_data_dictionary(trx) (bug 19727).
 * row0sel.c
   row_fetch_print(): Handle SQL NULL values without crashing.
   row_sel_store_mysql_rec(): Remove useless call to rec_get_nth_field
   when handling an externally stored column.
   Fetch externally stored fields when using InnoDB's internal SQL
   parser.
   Optimize BLOB selects by using prebuilt->blob_heap directly instead
   of first reading BLOB data to a temporary heap and then copying it
   to prebuilt->blob_heap.
 * srv0srv.c
   srv_master_thread(): Remove unreachable code.
 * srv0start.c
   srv_parse_data_file_paths_and_sizes(): Accept lower-case 'm' and
   'g' as abbreviations of megabyte and gigabyte (bug 19609).
   srv_parse_megabytes(): New fuction.
 * ut0dbg.c|h
   Implement InnoDB assertions (ut_a and ut_error) with abort() when
   the code is compiled with GCC 3 or later on other platforms than
   Windows or Netware. Also disable the variable ut_dbg_stop_threads
   and the function ut_dbg_stop_thread() i this case, unless
   UNIV_SYC_DEBUG is defined. This should allow the compiler to
   generate more compact code for assertions.
 * ut0list.c|h
   Add ib_list_create_heap().

Restoring default innodb_thread_concurrency to 8
This default now matches 5.0

(Fixes core dump in rpl_failed_optimize.test)
Ensure we end active transcations if we do an admin command (like optimize, repair etc)

* Fix BUG#15650: "DELETE with LEFT JOIN crashes server with innodb_locks_unsafe_for binlog"
* Fix BUG#17134: "Partitions: uncommitted changes are visible"
* Fix BUG#17992: "Partitions: InnoDB, somehow rotten table after UPDATE"
  row0ins.c: MySQL's partitioned table code does not set preduilt->sql_stat_start right
  if it does an insert in the same statement after doing a search first in the same
  partition table. We now write trx id always to the buffer, not just when flag
  sql_stat_start is on. This will waste CPU time very sightly.
* Fix BUG#18077: "InnoDB uses full explicit table locks in stored FUNCTION"
* Fix BUG#18238: "When locks exhaust the buffer pool, InnoDB does not roll back the trx"
* Fix BUG#18252" "Disk space leak in updates of InnoDB BLOB rows in 5.0 and 5.1"
* Fix BUG#18283: "When InnoDB returns error 'lock table full', MySQL can write to binlog too much"
* Fix BUG#18350: "Use consistent read in CREATE ... SELECT ... if innodb_locks_unsafe_for_binlog"
* Fix BUG#18384: "InnoDB memory leak on duplicate key errors in 5.0 if row has many columns"
* Fix BUG#18934: "InnoDB crashes when table uses column names like DB_ROW_ID"
  Refuse tables that use reserved column names.
* InnoDB's SQL parser:
  - Add support for UNSIGNED types, EXIT keyword, quoted identifiers, user-function callbacks
    for processing results of FETCH statements, bound literals, DATA_VARCHAR for bound literals.
  - Allow bound literals of type non-INTEGER to be of length 0.
  - Add make_flex.sh and update lexer/parser generation documentation.
  - Add comment clarifying the difference between 'alias' and 'indirection' fields in sym_node_t.
  - Remove never reached duplicate code in pars_set_dfield_type().
  - Rewrite pars_info datatypes and APIs, add a few helper functions.
  - Since the functions definitions in pars_info_t are accessed after pars_sql() returns
    in the query graph execution stage, we can't free pars_info_t in pars_sql(). Instead,
    make pars_sql() transfer ownership of pars_info_t to the created query graph, and
    make que_graph_free() free it if needed.
  - Allow access to system columns like DB_ROW_ID.
* Use bound literals in row_truncate_table_for_mysql, row_drop_table_for_mysql,
  row_discard_tablespace_for_mysql, and row_rename_table_for_mysql.
* Setting an isolation level of the transaction to read committed weakens the locks for
  this session similarly like the option innodb_locks_unsafe_for binlog. This patch removes
  alnost all gap locking (used in next-key locking) and makes MySQL to release the row locks
  on the rows which does not belong to result set. Additionally, nonlocking selects on
  INSERT INTO SELECT, UPDATE ... (SELECT ...), and CREATE ... SELECT ... use a nonlocking
  consistent read. If a binlog is used, then binlog format should be set to row based
  binloging to make the execution of the complex SQL statements.
* Disable the statistic variables btr_search_n_hash_fail and n_hash_succ, n_hash_fail,
  n_patt_succ, and n_searches of btr_search_t in builds without #ifdef UNIV_SEARCH_PERF_STAT.
* Make innodb.test faster. Group all consistent read test cases to a one test case and
  wait their lock timeout after all have been send to the server. Decrease amount of rows
  inserted in a certain test - this has no effect on the effectiveness of the test and
  reduces the running time by ~10 sec. Remove temporary work-arounds from innodb.result
  now that ALTER TABLE DROP FOREIGN KEY works once again.
* Make innodb_unsafe_binlog.test faster. Grout all consistent read test cases to a one
  test case amd wait their lock timeout after all have been sent to the server. Remove
  unnecessary option --loose_innodb_lock_wait_timeout.
* Print dictionary memory size in SHOW INNODB STATUS.
* Fix memory leaks in row_create_table_for_mysql() in rare corner cases.
* Remove code related to clustered tables. They were never implemented, and the
  implementation would be challenging with ROW_FORMAT=COMPACT. Remove the table types
  DICT_TABLE_CLUSTER_MEMBER and DICT_TABLE_CLUSTER and all related tests and functions.
  dict_table_t: Remove mix_id, mix_len, mix_id_len, mix_id_buf, and cluster_name.
  plan_t: Remove mixed_index.
  dict_create_sys_tables_tuple(): Set MIX_ID=0, MIX_LEN=0, CLUSTER_NAME=NULL when
  inserting into SYS_TABLES.
  dict_tree_check_search_tuple(): Enclose in #ifdef UNIV_DEBUG.
* Move calling of thr_local_free() from trx_free_for_mysql() to
  innobase_close_connection().

 the innodb-5.0-ss476 snapshot.

 Fix BUG#18934: "InnoDB crashes when table uses column like DB_ROW_ID".
 Also, fix memory leaks in row_create_table_for_mysql() in rare
 corner cases.

Without a flush this test is nondeterministic for "row" binlog-format.

  Testsuite added

InnoDB requires a full table rebuild for foreign key changes.
It was not possible in compare_tables() to detect such changes.

On Heikkis proposal I added a new flag to the syntax parser
where foreign key definition changes are done. I test for
this flag in compare_tables() now.

  Fixed BUGS:
  #3300: "UPDATE statement with no index column in where condition locks
    all rows"
    Implement semi-consistent read to reduce lock conflicts at the cost
    of breaking serializability.
    ha_innobase::unlock_row(): reset the "did semi consistent read" flag
    ha_innobase::was_semi_consistent_read(),
    ha_innobase::try_semi_consistent_read(): new methods
    row_prebuilt_t, row_create_prebuilt(): add field row_read_type for
    keeping track of semi-consistent reads
    row_vers_build_for_semi_consistent_read(),
    row_sel_build_committed_vers_for_mysql(): new functions
    row_search_for_mysql(): implement semi-consistent reads

  #9802: "Foreign key checks disallow alter table".
    Added test cases.

  #12456: "Cursor shows incorrect data - DML does not affect,
    probably caching"
    This patch implements a high-granularity read view to be used with
    cursors. In this high-granularity consistent read view modifications 
    done by the creating transaction after the cursor is created or 
   future transactions are not visible. But those modifications that 
   transaction did before the cursor was created are visible.

  #12701: "Support >4GB buffer pool and log files on 64-bit Windows"
    Do not call os_file_create_tmpfile() at runtime. Instead, create all
    tempfiles at startup and guard access to them with mutexes.

  #13778: "If FOREIGN_KEY_CHECKS=0, one can create inconsistent FOREIGN KEYs".
    When FOREIGN_KEY_CHECKS=0 we still need to check that datatypes between
    foreign key references are compatible.

  #14189: "VARBINARY and BINARY variables: trailing space ignored with InnoDB"
    innobase_init(): Assert that
    DATA_MYSQL_BINARY_CHARSET_COLL == my_charset_bin.number.
    dtype_get_pad_char(): Do not pad VARBINARY or BINARY columns.
    row_ins_cascade_calc_update_vec(): Refuse ON UPDATE CASCADE when trying
    to change the length of a VARBINARY column that refers to or is referenced
    by a BINARY column. BINARY columns are no longer padded on comparison,
    and thus they cannot be padded on storage either.

  #14747: "Race condition can cause btr_search_drop_page_hash_index() to crash"
    Note that buf_block_t::index should be protected by btr_search_latch
    or an s-latch or x-latch on the index page.
    btr_search_drop_page_hash_index(): Read block->index while holding
    btr_search_latch and use the cached value in the loop.  Remove some
    redundant assertions.

  #15108: "mysqld crashes when innodb_log_file_size is set > 4G"

  #15308: "Problem of Order with Enum Column in Primary Key"

  #15550: "mysqld crashes in printing a FOREIGN KEY error in InnoDB"
    row_ins_foreign_report_add_err(): When printing the parent record,
    use the index in the parent table rather than the index in the child table.

  #15653: "Slow inserts to InnoDB if many thousands of .ibd files"
    Keep track on unflushed modifications to file spaces.  When there are tens
    of thousands of file spaces, flushing all files in fil_flush_file_spaces()
    would be very slow.
    fil_flush_file_spaces(): Only flush unflushed file spaces.
    fil_space_t, fil_system_t: Add a list of unflushed spaces.

  #15991: "innodb-file-per-table + symlink database + rename = cr"
   os_file_handle_error(): Map the error codes EXDEV, ENOTDIR, and EISDIR
   to the new code OS_FILE_PATH_ERROR. Treat this code as OS_FILE_PATH_ERROR.
   This fixes the crash on RENAME TABLE when the .ibd file is a symbolic link
   to a different file system.

  #16157: "InnoDB crashes when main location settings are empty"
    This patch is from Heikki.

  #16298: "InnoDB segfaults in INSERTs in upgrade of 4.0 -> 5.0 tables
    with VARCHAR BINARY"
    dict_load_columns(): Set the charset-collation code
    DATA_MYSQL_BINARY_CHARSET_COLL for those binary string columns
    that lack a charset-collation code, i.e., the tables were created
    with an older version of MySQL/InnoDB than 4.1.2.

  #16229: "MySQL/InnoDB uses full explicit table locks in trigger processing"
    Take a InnoDB table lock only if user has explicitly requested a table
    lock. Added some additional comments to store_lock() and external_lock().

  #16387: "InnoDB crash when dropping a foreign key <table>_ibfk_0"
    Do not mistake TABLENAME_ibfk_0 for auto-generated id.
    dict_table_get_highest_foreign_id(): Ignore foreign constraint
    identifiers starting with the pattern TABLENAME_ibfk_0.

  #16582: "InnoDB: Error in an adaptive hash index pointer to page"
    Account for a race condition when dropping the adaptive hash index
    for a B-tree page.
    btr_search_drop_page_hash_index(): Retry the operation if a hash index
    with different parameters was built meanwhile.  Add diagnostics for the
    case that hash node pointers to the page remain.
    btr_search_info_update_hash(), btr_search_info_update_slow():
    Document the parameter "info" as in/out.

  #16814: "SHOW INNODB STATUS format error in LATEST FOREIGN KEY ERROR
    section"
    Add a missing newline to the LAST FOREIGN KEY ERROR section in SHOW
    INNODB STATUS output.
    dict_foreign_error_report(): Always print a newline after invoking
    dict_print_info_on_foreign_key_in_create_format().

  #16827: "Better InnoDB error message if ibdata files omitted from my.cnf"

  #17126: "CHECK TABLE on InnoDB causes a short hang during check of adaptive
    hash"
    CHECK TABLE blocking other queries, by releasing the btr_search_latch
    periodically during the adaptive hash table validation.

  #17405: "Valgrind: conditional jump or move depends on unititialised values"
    buf_block_init(): Reset magic_n, buf_fix_count and io_fix to avoid
    testing uninitialized variables.

(Needed for "list of pushes" web page and autopush)

  Applied 9680.patch (by Osku Salerma)

There are (at least) two implementations of the checksum
computation. One is in MyISAM for the quick checksum. It
is executed on every row change. The other is in the
SQL layer for the extended checksum. It retrieves all rows
of a table via the respective storage engine.

In former MySQL versions varchars were stored with their 
maximum length, but now with their real length similar to
blobs.

This change had been forgotten to take care of in the
extended checksum calculation. Hence too much data was
checksumed. In MyISAM this change had been taken care of 
already. Only the real data is included in the checksum.

I changed mysql_checksum_table() so that it uses the
length information of true varchar fields instead
of the field length like in former varchar 
implementations.

  Applied innodb-5.0-ss149/162 snapshots.

  Applied innodb-4.1-ss17 snapshot.
  Do not mistake TABLENAME_ibfk_0 for auto-generated id.

Change "duplicate key" message to print key name
instead of key number.

 Make innodb_thread_concurrency 0 by default, and extend the usable
 range from 0 to 1000 (0 disables the thread throttling).

 Fixed bugs:
 BUG#15991: "innodb-file-per-table + symlink database + rename = crash"
 BUG#15650: "DELETE with LEFT JOIN crashes server"
 BUG#15308: "Problem of Order with Enum Column in Primary Key"
 BUG#14189: "VARBINARY and BINARY variables: trailing space ignored"

 Fixed BUG#14056: Column prefix index on UTF-8 primary key
 causes "Can't find record.."
 Also fixed bug 15991.

This includes both code and test cases.

 table' lockup".
 Changes from the innodb-5.0-ss92 snapshot.
 Do not call os_file_create_tmpfile() at runtime. Instead, create
 all tempfiles at startup and guard access to them with mutexes.

 Removed include/Makefile.am and the reference to it.
 Deleted db/db0err.h and db directory.
 Check index column sizes in a better way (bug 13315).
 Fixed comments for memory allocation functions and added
 some extra checks. Adapted callers.

 table' lockup".
 Changes from the innodb-4.1-ss11 snapshot.
 Do not call os_file-create_tmpfile() at runtime. Instead, create
 a tempfile at startup and guard access to it with a mutex.
 Also, fix bugs:
 10511: "Wrong padding of UCS2 CHAR columns in ON UPDATE CASCADE";
 13778: "If FOREIGN_KEY_CHECKS=0, one can create inconsistent FOREIGN
 KEYs". When FOREIGN_KEY_CHECKS=0 we still need to check that
 datatypes between foreign key references are compatible.
 Also, added test cases (also for bug 9802).

 to 5.1. Undoing the merge.

 to crash".
 Changes from snapshot innodb-5.0-ss52.
 Note that buf_block_t::index should be protected by btr_search_latch
 or an s-latch or x-latch on the index page.
 btr_search_drop_page_hash_index(): Read block->index while holding
 btr_search_latch and use the cached value in the loop. Remove some
 redundant assertions.
 Also fix 13778. When FOREIGN_KEY_CHECKS=0 we still need to check that
 datatypes between foreign key references are compatible.
 Also added test cases to 9802.