Fixed lp:814237: Wrong mutex usage in Aria


storage/maria/ma_bitmap.c:
  Added call to _ma_bitmap_mark_file_changed() when flushing bitmap. This fixes lp:814237

scripts/mysqldumpslow.sh:
  Remove not used --basedir option
  Ensure that the last --datadir option is used from the my.cnf files.

The bug happens when one uses MAX_ROWS=# with Aria & row_format=page and one insert more than # rows.


mysql-test/mysql-test-run.pl:
  Ignore table is full error messages
mysql-test/suite/maria/r/max_length.result:
  Test case for 'Table is full'
mysql-test/suite/maria/t/max_length.test:
  Test case for 'Table is full'
storage/maria/ma_bitmap.c:
  Ensure that we don't allocate bits outside of max_data_file_size.
  Adjust max_data_file_size based on bitmap alignments.
  Backport fix to adjust wrong first_bitmap_with_space.
storage/maria/ma_blockrec.c:
  Calculate value of max_data_file_length
storage/maria/ma_blockrec.h:
  Updated prototype for _ma_bitmap_init()
storage/maria/ma_check.c:
  Give warnings if file sizes are above max file sizes.
  Give more warnings in case of errors.
  Have maria_chk write if table is recreated.
storage/maria/ma_create.c:
  Better calculation of max_data_file_length and thus data pointer length.
  Fixes some wrong pointer lengths when using MAX_ROWS=#
storage/maria/ma_open.c:
  Removed duplicate assigment.
  Use block size from file instead of global variable.
storage/maria/maria_chk.c:
  Remove -1 from printed file length
storage/maria/maria_def.h:
  Update struct st_maria_file_bitmap

Fixed explains of previous patch.

mysql-test/r/explain.result:
  Fixed explains of previous patch.
mysql-test/r/join_outer.result:
  Fixed explains of previous patch.
mysql-test/r/negation_elimination.result:
  Fixed explains of previous patch.
mysql-test/r/view.result:
  Fixed explains of previous patch.
mysql-test/suite/innodb/r/innodb_mysql.result:
  Removed duplicate test suite.
mysql-test/suite/innodb/t/innodb_mysql.test:
  Removed duplicate test suite.
mysql-test/suite/innodb_plugin/r/innodb_mysql.result:
  Removed duplicate test suite.
mysql-test/suite/innodb_plugin/t/innodb_mysql.test:
  Removed duplicate test suite.
sql/opt_range.h:
  Removed incorrect fix.
sql/records.cc:
  Removed incorrect fix.

There are 2 volatile condition constructions AND/OR constructions and fields(references) when first
good supported to be top elements of conditions because it is normal practice
(see copy_andor_structure for example) fields without any expression in the condition is really rare
and mostly useless case however it could lead to problems when optimiser changes/moves them unaware
of other variables referring to them. An easy solution of this problem is just to replace single field
in a condition with equivalent expression well supported by the server (<field> -> <field> != 0).

mysql-test/r/view.result:
  New test added.
mysql-test/t/view.test:
  New test added.
sql/sql_parse.cc:
  <field> -> <field> != 0
sql/sql_yacc.yy:
  <field> -> <field> != 0

Fix is to replace uint in public header with unsigned int.  uint is not guaranteed to be defined by system headers.

before strlen(db) we need to be sure that
db lies within packet boundaries.
same for client_plugin.

before strlen(db) we need to be sure that
db lies within packet boundaries

- Fixed multi-user problem with one thread doing inserts and another doing scans that gave error 175
- Fixed bug that caused assert in move_to_next_bitmap() & _ma_read_bitmap_page()
- Much more DBUG_ASSERT(!maria_assert_if_crashed_table) to detect errors early
- EXTERNAL_LOCKING -> MARIA_EXTERNAL_LOCKING (to use same define everywhere

storage/maria/ma_bitmap.c:
  More secure handling of first_bitmap_with_space (now we also take care of wrong values)
  Don't set page to -1; This fixed unlikely bug that caused assert in move_to_next_bitmap() & _ma_read_bitmap_page()
storage/maria/ma_blockrec.c:
  More DBUG_ASSERT()'s
  Fixed multi-user problem with one thread doing inserts and another doing scans that gave error 175
  (We should use data_file_length from start of scan, not new value as new bitmap page may not yet be in page cache)
storage/maria/ma_check.c:
  EXTERNAL_LOCKING -> MARIA_EXTERNAL_LOCKING (to use same define everywhere)
storage/maria/ma_checkpoint.c:
  Made maria_checkpoint_min_activity static so that one can change it in debugger while testing.
  Fixed long standing performance problem that caused write of state info at checkpoint for any file that was ever changed since open.
storage/maria/ma_create.c:
  EXTERNAL_LOCKING -> MARIA_EXTERNAL_LOCKING (to use same define everywhere
storage/maria/ma_dynrec.c:
  Added missing MARIA_EXTERNAL_LOCKING (minor performance improvement)
storage/maria/ma_locking.c:
  EXTERNAL_LOCKING -> MARIA_EXTERNAL_LOCKING (to use same define everywhere
storage/maria/ma_open.c:
  EXTERNAL_LOCKING -> MARIA_EXTERNAL_LOCKING (to use same define everywhere
storage/maria/ma_pagecache.c:
  Added assert to detect reads outside of data file
storage/maria/maria_def.h:
  Added checkpoint_state to cache state writes in checkpoint

- Fixed performance bug in alter table with Aria; Aria didn't use disable keys + enable keys
- Fixed wrong warning about 'Wrong CRC on datapage' from REPAIR TABLE with aria block tables.
- Fixed bug in aria_chk that disabled performance counters.
- Added --translog_buffer_size to maria_read_log.

storage/maria/ha_maria.cc:
  Fixed performance bug in alter table with Aria; Aria didn't use disable keys + enable keys
storage/maria/ma_bitmap.c:
  Added some DBUG_ASSERT()'s and made code more uniform
storage/maria/ma_check.c:
  Fixed wrong warning about 'Wrong CRC on datapage' from REPAIR TABLE with aria block tables.
storage/maria/ma_pagecache.c:
  Copy read pages to buffer even if there was an error (to be able to detect zero filled pages)
storage/maria/maria_chk.c:
  Fixed bug in aria_chk that disabled performance counters.
storage/maria/maria_read_log.c:
  Added option to set translog_buffer_size

Fixed several errors in Aria discovered by test case for lp:727869 ma_pagecache.c:2103: find_block: Assertion `block->rlocks == 0
- Fixed assert in transaction log handler when aria_check was run on block-record table that was much bigger than expected.
- Fixed warnings about wrong mutex order between bitmap and intern_lock
- Fixed error in bitmap that could cause two rows to use same block for a block record.
- Fixed wrong test that could cause error if last page for a bitmap was used by a blob.
- Fixed several bugs in pagecache for the case where pagecase had very few blocks and there was a lot of threads competing to get the blocks (very unlikely case).


mysql-test/suite/maria/r/maria-recovery3.result:
  Updated results
sql/mysqld.cc:
  Allow mi_check() to send information messages for log file
storage/maria/ma_bitmap.c:
  Fixed problem with wrong mutex order when bitmap was the first page that was flushed out of page cache
  - Fixed by introducing _ma_bitmap_mark_file_changed() that marks file changed without a bitmap lock.
  - Fixed one case in _ma_change_bitmap_page() where we didn't mark the bitmap changed. This could cause to rows to reuse same block if this was the only change to the bitmap.
  - Split _ma_bitmap_get_page_bits() in two parts to not take a bitmap lock when we already have it
  - Fixed bug in _ma_bitmap_set_full_page_bits() that caused an error if last page for a bitmap was used by a blob
storage/maria/ma_check.c:
  Better handling of wrong file length.
  Fixed bug when we tried to write to transaction log when it was not opened (happened when block record file was bigger than expected)
storage/maria/ma_pagecache.c:
  Fixed several bugs in pagecache for the case where pagecase had very few blocks and there was a lot of threads competing to get the blocks:
  - In link_block() mark a block given to another thread with PCBLOCK_REASSIGNED to ensure that no other threads can start re-using the block
    before the thread that requsted a block.
  - In free_block(), don't reset status for a block that is in re-assign by link_block() (we don't want to loose the PCBLOCK_REASSIGNED flag).
  - Added call to wait_for_flush() when we got a new block in find_block() to ensure that we don't use a block that is beeing flushed by another thread.
  - Moved setting of hits_left and last_hit_time in find_block() to where we assign the block.
  
  
  Code cleanup and making code uniform:
  - Changed a lot of KEYCACHE_DBUG_PRINT to use DBUG_PRINT
  - Streamlined all reporting of 'signal' and 'wait' between threads to be identical.
  - Use thread name instead of thread number (for each match against --debug)
  - Added more DBUG_ENTER, DBUG_PRINT and DBUG_ASSERT()
  - Added more comments
storage/myisam/ha_myisam.cc:
  Only print information about that we make a backup if we are really making a backup.
storage/myisam/mi_check.c:
  Inform mysqld that we are creating a backup of the data file (for inclusion in error log).

- Fixed error when writing a blob to the last page on the bitmap.
- Marked bitmap changed in once case that could cause two rows to use the same blob page.

mysql-test/r/join.result:
  Test case for LP:798597
mysql-test/t/join.test:
  Test case for LP:798597
sql/sql_select.cc:
  In simplify_joins we reset table->maybe_null for outer join tables that can't ever be NULL.
  This caused a conflict between the previously calculated items and the group_buffer against the fields
  in the temporary table that are created as not null thanks to the optimization.
  The fix is to correct the group by items to also be not_null so that they match the used fields and keys.

initialize plugins earlier, to support, for example, non-MyISAM mysql.plugin table.

Define USE_MARIA_FOR_TMP_TABLES preprocessor constant by default if Aria engine is compiled in.
Use CMake variable WITH_ARIA_TMP_TABLES to control the temp table engine setting.

(without CFLAGS=-DSAFEMALLOC).

Added option to aria_read_log to crash recovery at certain points in the recovery process.
Fixed bug that caused future recovery attempts to fail if we got a crash/got killed during closing of tables at end of recovery process.


mysql-test/mysql-test-run.pl:
  Don't abort if 'var' points to stale place; Just remove it.
mysql-test/suite/maria/r/maria.result:
  Fixed wrong indentation
mysql-test/t/partition_error.test:
  Fixed portability problem with partiton_error.test
storage/maria/ma_close.c:
  More DBUG_PRINT info
storage/maria/ma_pagecache.c:
  Copy flush_log_callback when writing to page cache. This fixes problem in recovery when switching from mode of file
storage/maria/ma_recovery.c:
  Added option to aria_read_log to crash recovery at certain points in the recovery process.
storage/maria/ma_recovery.h:
  Added option to aria_read_log to crash recovery at certain points in the recovery process.
storage/maria/maria_chk.c:
  Align aria_chk -d output
  Don't write warning Aria table '...' is usable but should be fixed if the table was before marked as crashed but now is ok
storage/maria/maria_read_log.c:
  Added option to aria_read_log to crash recovery at certain points in the recovery process.

Noted that there was no memory leak, just a lot of used partitioned tables.
Fixed old bug: 'show status' now shows memory usage when compiled with safemalloc.
Added option --flush to mysqlcheck.c to run a 'flush tables' between each check to keep down memory usage.
Changed '--safemalloc' options to mysqld so that one can use --safemalloc and --skip-safemalloc.
Now skip-safemalloc is default (ie, we only do checking of memory overrun during free()) to speed up tests.


client/client_priv.h:
  Added OPT_FLUSH_TABLES
client/mysqlcheck.c:
  Added option --flush to mysqlcheck.c to run a 'flush tables' between each check to keep down memory usage.
mysql-test/mysql-test-run.pl:
  Always run tests with --loose-skip-safemysqld for higher speed
sql/mysqld.cc:
  Changed '--safemalloc' options so that one can use --safemalloc and --skip-safemalloc.
  Now skip-safemalloc is default (ie, we only do checking of memory overrun during free()) to speed up tests
sql/sql_parse.cc:
  Fixed old bug: 'show status' now shows memory usage when compiled with safemalloc.
storage/archive/archive_reader.c:
  Changed all malloc() calls to use my_malloc()/my_free()
  Added checks of malloc() calls.
storage/archive/ha_archive.cc:
  Detect failure if init_archive_reader() and return errno. This fixed assert crash in my_seek().
  Changed all malloc() calls to use my_malloc()/my_free()