An error occurred fetching the project authors.
  1. 10 May, 2013 1 commit
    • Michael Widenius's avatar
      Added some fixes that should make MyISAM & Aria REPAIR work with more than 4G records · 53d44ad1
      Michael Widenius authored
      - If one specifies --force twice to myisamchk and aria_chk, then we will try to finnish the repair even if sort_buffer would be too small.
        This was done by dynamically allocate buffer handler objects as long as memory lasts.
      - New option for myisamchk and aria_chk: create-missing-keys
      - Changed default size of myisam_sort_buffer_size from 8M to 128M.
      - Changed default size of sort_buffer_size in aria_chk from 128M to 256M.
      - Increased information in error message about 'sort_buffer_size' beeing to small.
      - Print also to 'show warnings' if repair was retried.
      - Increased size of internal sort-buffer-readers from 16K to 128K
      - Changed printing of 'number of records' to use %ll instead of casting to long
      - Changed buffer sizes for myisam and aria to use MY_ALIGN_DOWN() to get same number of bytes allocated on different machines.
      
      
      
      
      
      include/my_global.h:
        Added MY_ALIGN_DOWN() to get previous alignment (for big memory areas)
      include/myisam.h:
        Increased size of types to be able to handle more records
      include/myisamchk.h:
        Increased size of types to be able to handle more records
        Added T_FORCE_SORT_MEMORY to force repair to work even if sort_buffer would not be big enough
      mysql-test/r/myisam.result:
        Updated result
      mysql-test/r/mysqld--help.result:
        Updated result
      mysql-test/r/repair.result:
        Updated result
      mysql-test/suite/maria/maria.result:
        Added test cases for some fixed bugs in MyISAM to verify that Aria doesn't have them too.
      mysql-test/suite/maria/maria.test:
        Added test cases for some fixed bugs in MyISAM to verify that Aria doesn't have them too.
      mysql-test/suite/maria/maria3.result:
        Updated result after sort buffer size increase
      mysql-test/suite/sys_vars/r/aria_sort_buffer_size_basic.result:
        Updated result after sort buffer size increase
      mysql-test/suite/sys_vars/r/myisam_sort_buffer_size_basic.result:
        Updated result after sort buffer size increase
      mysql-test/suite/sys_vars/t/myisam_sort_buffer_size_basic.test:
        Updated result after sort buffer size increase
      mysql-test/t/myisam.test:
        Fixed error messages to not print system specific data
      mysql-test/t/repair.test:
        Fixed error messages to not print system specific data
      storage/maria/ha_maria.cc:
        Print also to 'show warnings' if repair was retried
        Changed default size of sort_buffer_size from 128M to 256M (same as in mysqld)
      storage/maria/ma_check.c:
        Renamed USE_BUFFER_INIT -> PAGE_BUFFER_INIT
      storage/maria/ma_sort.c:
        Increased size of internal sort-buffer-readers from 16K to 128K
        Increased size of types to be able to handle more records
        Added support for T_FORCE_SORT_MEMORY
        Don't allocate too many extra BUFFPEK at a time (they are probably not needed)
        Improved error message for "sort_buffer_size is too small"
        Changed printing of 'number of records' to use %ll instead of casting to long
        Fixed bug where maria_update_key_parts() was called too early.
        Fixed bug in detecting result from read_to_buffer().
        Added 'out of memory' checking when calling 'alloc_dynamic()'.
      storage/maria/maria_chk.c:
        Added --create-missing-keys
        If one specifies --force twice then we will try to finnish the repair even if sort_buffer would be too small.
        check_param.sort_buffer_length varialble was used with wrong type.
      storage/maria/maria_def.h:
        Increased size of types to be able to handle more records
        Use MY_ALIGN_DOWN() to get same number of bytes allocated on different machines
        Renamed USE_BUFFER_INIT -> PAGE_BUFFER_INIT
      storage/maria/maria_ftdump.c:
        Renamed USE_BUFFER_INIT -> PAGE_BUFFER_INIT
      storage/maria/maria_read_log.c:
        Use PAGE_BUFFER_INIT for page cache
      storage/myisam/ha_myisam.cc:
        Changed default size of myisam_sort_buffer_size from 8M to 128M
      storage/myisam/mi_check.c:
        Renamed USE_BUFFER_INIT -> KEY_BUFFER_INIT
      storage/myisam/myisam_ftdump.c:
        Renamed USE_BUFFER_INIT -> KEY_BUFFER_INIT
      storage/myisam/myisamchk.c:
        Added --create-missing-keys
        If one specifies --force twice then we will try to finnish the repair even if sort_buffer would be too small.
        check_param.sort_buffer_length varialble was used with wrong type.
        Renamed USE_BUFFER_INIT -> KEY_BUFFER_INIT
      storage/myisam/myisamdef.h:
        Increased SORT_BUFFER_INIT to 64M
        (speeds up repair a lot and most machines have nowadays a lot of memory)
        Use MY_ALIGN_DOWN() to get same number of bytes allocated on different machines
        Renamed USE_BUFFER_INIT -> KEY_BUFFER_INIT
      storage/myisam/sort.c:
        Increased size of internal sort-buffer-readers from 16K to 128K
        Increased size of types to be able to handle more records
        Added support for T_FORCE_SORT_MEMORY
        Don't allocate too many extra BUFFPEK at a time (they are probably not needed)
        Improved error message for "sort_buffer_size is too small"
        Changed printing of 'number of records' to use %ll instead of casting to long
        Fixed bug in detecting result from read_to_buffer().
        Added 'out of memory' checking when calling 'alloc_dynamic()'.
      53d44ad1
  2. 29 Jun, 2011 1 commit
    • Michael Widenius's avatar
      Aria bug fixes and improvements: · 5e876bd3
      Michael Widenius authored
      - Fixed performance bug in alter table with Aria; Aria didn't use disable keys + enable keys
      - Fixed wrong warning about 'Wrong CRC on datapage' from REPAIR TABLE with aria block tables.
      - Fixed bug in aria_chk that disabled performance counters.
      - Added --translog_buffer_size to maria_read_log.
      
      storage/maria/ha_maria.cc:
        Fixed performance bug in alter table with Aria; Aria didn't use disable keys + enable keys
      storage/maria/ma_bitmap.c:
        Added some DBUG_ASSERT()'s and made code more uniform
      storage/maria/ma_check.c:
        Fixed wrong warning about 'Wrong CRC on datapage' from REPAIR TABLE with aria block tables.
      storage/maria/ma_pagecache.c:
        Copy read pages to buffer even if there was an error (to be able to detect zero filled pages)
      storage/maria/maria_chk.c:
        Fixed bug in aria_chk that disabled performance counters.
      storage/maria/maria_read_log.c:
        Added option to set translog_buffer_size
      5e876bd3
  3. 13 Jun, 2011 1 commit
    • Michael Widenius's avatar
      Fixed portability problem with partiton_error.test · bf0e1f44
      Michael Widenius authored
      Added option to aria_read_log to crash recovery at certain points in the recovery process.
      Fixed bug that caused future recovery attempts to fail if we got a crash/got killed during closing of tables at end of recovery process.
      
      
      mysql-test/mysql-test-run.pl:
        Don't abort if 'var' points to stale place; Just remove it.
      mysql-test/suite/maria/r/maria.result:
        Fixed wrong indentation
      mysql-test/t/partition_error.test:
        Fixed portability problem with partiton_error.test
      storage/maria/ma_close.c:
        More DBUG_PRINT info
      storage/maria/ma_pagecache.c:
        Copy flush_log_callback when writing to page cache. This fixes problem in recovery when switching from mode of file
      storage/maria/ma_recovery.c:
        Added option to aria_read_log to crash recovery at certain points in the recovery process.
      storage/maria/ma_recovery.h:
        Added option to aria_read_log to crash recovery at certain points in the recovery process.
      storage/maria/maria_chk.c:
        Align aria_chk -d output
        Don't write warning Aria table '...' is usable but should be fixed if the table was before marked as crashed but now is ok
      storage/maria/maria_read_log.c:
        Added option to aria_read_log to crash recovery at certain points in the recovery process.
      bf0e1f44
  4. 25 Apr, 2011 1 commit
  5. 25 Feb, 2011 1 commit
    • Michael Widenius's avatar
      - maria/ma_test_all.sh can now be run with --tmpdir=/dev/shm for faster testing · 07b5d554
      Michael Widenius authored
      - Fixed mysql-test-run failures on window
      - Fixed compiler warnings from my last push (sorry about that)
      - Fixed that maria_chk --repair --extended works again
      - Fixed compiler warnings about using not unitialized data
      
      mysql-test/mysql-test-run.pl:
        Better output
      mysql-test/suite/parts/inc/partition_check_drop.inc:
        Use remove_files_wildcard instead of rm
      mysys/safemalloc.c:
        Fixed argument to printf
      storage/maria/ma_cache.c:
        Don't give errors when running maria_chk
      storage/maria/ma_dynrec.c:
        Don't give errors when running maria_chk
      storage/maria/ma_rt_test.c:
        Added option --datadir for where to put logs and test data
      storage/maria/ma_test1.c:
        Added option --datadir for where to put logs and test data
      storage/maria/ma_test2.c:
        Added option --datadir for where to put logs and test data
      storage/maria/maria_chk.c:
        If --datadir is used but --logdir is not, set --logdir from --datadir
        (this reflects how --help said how things should work)
      storage/maria/maria_read_log.c:
        Changed short option for 'maria-log-dir-path' from -l to -h to be same as mysqld, maria_chk, ma_test1 etc..
      storage/maria/unittest/ma_test_all-t:
        Allow one to specify --tmpdir for where to store logs and data
      storage/xtradb/buf/buf0buf.c:
        Fixed compiler warnings about using not unitialized data
      storage/xtradb/row/row0upd.c:
        Fixed compiler warnings about using not unitialized data
      storage/xtradb/srv/srv0srv.c:
        Fixed compiler warnings about using not unitialized data
      07b5d554
  6. 11 Jan, 2011 2 commits
  7. 06 Jan, 2011 1 commit
    • Michael Widenius's avatar
      Bug fixing in Aria: · 5b0afd8a
      Michael Widenius authored
      - Fixed some bugs in recovery of blobs
      - Don't ASSERT() on checksum errors when running check table
      - Added to maria_read_log option --tables-to-redo=list-of-tables to only recover some tables (good for debugging)
      
      
      storage/maria/ma_blockrec.c:
        Don't ASSERT() on checksum errors when running check table
        Fixed bug in recovery of blog page that was not in dirty pages list
      storage/maria/ma_check.c:
        Don't ASSERT() on checksum errors when running check table
      storage/maria/ma_recovery.c:
        Handling of --tables-to-redo
      storage/maria/ma_recovery.h:
        Handling of --tables-to-redo
      storage/maria/ma_recovery_util.c:
        Give better warning if table was not in dirty pages list
      storage/maria/maria_def.h:
        Added in_check_table
      storage/maria/maria_read_log.c:
        Added --tables-to-redo=list-of-tables to only recover some tables (good for debugging)
        Cleaned up message when wrong arguments
      5b0afd8a
  8. 12 Sep, 2010 1 commit
  9. 18 Aug, 2010 1 commit
    • Michael Widenius's avatar
      Added verbose mode to recovery · 909e4eb0
      Michael Widenius authored
      More DBUG
      Added convert-debug-for-diff
      Added missing (from last push) federated test case
      
      .bzrignore:
        Ignore generated files
      mysql-test/README:
        Updated documentation for --extern
      mysql-test/suite/federated/federated_bug_32426.result:
        Added test for federatedx
      mysql-test/suite/federated/federated_bug_32426.test:
        Added test for federatedx
      scripts/Makefile.am:
        Added convert-debug-for-diff
      scripts/convert-debug-for-diff.sh:
        Added script for converting
      sql/log.cc:
        Added DBUG
      sql/mysqld.cc:
        Added DBUG
      storage/maria/ma_recovery.c:
        If verbose, write dirty pages (for debugging)
      storage/maria/ma_static.c:
        Added verbose mode to recovery
      storage/maria/maria_def.h:
        Added verbose mode to recovery
      storage/maria/maria_read_log.c:
        Added verbose mode to recovery
      909e4eb0
  10. 12 Aug, 2010 1 commit
    • Michael Widenius's avatar
      Added option --start-from-checkpoint to maria_read_log · 6bbb0b57
      Michael Widenius authored
      Print out checked file names in maria_check -s (unless you use a second -s)
      Some trivial optimizations
      
      storage/maria/ma_bitmap.c:
        Trivial optimizations:
        - Combine common code (to be able to remove duplicate mutex_lock call)
        - Move setting of thread specific variables outside of mutex
      storage/maria/ma_check.c:
        Fixed wrong argument to printf
      storage/maria/maria_chk.c:
        Print out checked table names unless -s -s
      storage/maria/maria_read_log.c:
        Added option --start-from-checkpoint (to help find bugs in checkpoints)
      6bbb0b57
  11. 10 Aug, 2010 1 commit
    • Michael Widenius's avatar
      Fixed LP#605798 RQG: Table corruption after Maria engine recovery - "Wrong data in bitmap" · e6cf286b
      Michael Widenius authored
      maria_chk & maria_read_log now reads block size from control file.
      
      
      
      mysql-test/suite/maria/r/maria.result:
        Updated results after trivial change of maria_chk's output
      storage/maria/ma_bitmap.c:
        More DBUG_PRINT
      storage/maria/ma_blockrec.c:
        Fixed bug that we didn't mark page full in bitmap if directory is full
      storage/maria/ma_check.c:
        Write out if directory is full for errors in bitmap
      storage/maria/ma_control_file.c:
        Don't give error for wrong block size if block size is 0
      storage/maria/maria_chk.c:
        Read block size from control file
        In case of -dvv, write also out bitmap information (good for debugging)
      storage/maria/maria_read_log.c:
        Read block size from control file
        Fixed that maria_read_log works with different page size than TRANSLOG_PAGE_SIZE
      e6cf286b
  12. 05 Aug, 2010 1 commit
    • Michael Widenius's avatar
      Remove not used files · f66d2500
      Michael Widenius authored
      Added option: --warning-for-wrong-transaction-id to maria_check
      
      
      mysql-test/r/innodb-use-sys-malloc.result:
        Remove not used files
      mysql-test/t/innodb-use-sys-malloc-master.opt:
        Remove not used files
      storage/maria/ma_recovery.c:
        Added support for --lsn-end
      storage/maria/ma_recovery.h:
        New prototype
      storage/maria/ma_write.c:
        Added logging of KEY_OP_CHECK for all possible keyblock splits.
        (Helps us find error in log file handling)
      storage/maria/maria_chk.c:
        Added option: --warning-for-wrong-transaction-id
        --update-status now resets open count if check succeds
      storage/maria/maria_read_log.c:
        Added option --end-lsn for only appling part of log (good for debugging)
        Changed options to use '-' instead of '_'
      f66d2500
  13. 04 Aug, 2010 1 commit
    • Michael Widenius's avatar
      Fixed compiler warnings · 1f5b93e7
      Michael Widenius authored
      Fixed some wrong test cases
      Fixed bug in null handling in XtraDB
      
      
      extra/comp_err.c:
        Fixed compiler warnings
      extra/my_print_defaults.c:
        Fixed compiler warnings
      mysql-test/suite/binlog/t/binlog_killed.test:
        Added support for timeouts
      mysql-test/suite/funcs_1/r/is_columns_is.result:
        Updated results (INNODB_SYS_TABLES had got new column)
      scripts/mysql_install_db.sh:
        Fixed typo
      sql/mysql_priv.h:
        Removed not needed argument for compare_record()
      sql/sql_insert.cc:
        Removed not needed argument for compare_record()
      sql/sql_update.cc:
        Removed not needed argument for compare_record()
        The argument is not needed becasue we copy the full record[0] to record[1] and the comparison should work even if all columns are not read
      sql/table.cc:
        The comparison of rows is independent of HA_PARTIAL_COLUMN_READ
      storage/maria/maria_chk.c:
        Fixed compiler warnings
      storage/maria/maria_read_log.c:
        Fixed compiler warnings
      storage/myisam/myisamchk.c:
        Fixed compiler warnings
      storage/myisam/myisampack.c:
        Fixed compiler warnings
      storage/xtradb/dict/dict0load.c:
        Fixed compiler warnings
      storage/xtradb/row/row0sel.c:
        Fixed null handling in XtraDB. (See comment)
      storage/xtradb/trx/trx0sys.c:
        Fixed compiler warnings
      support-files/compiler_warnings.supp:
        Fixed compiler warnings
      1f5b93e7
  14. 30 Jul, 2010 1 commit
    • Michael Widenius's avatar
      Fix for LP#602604: RQG: ma_blockrec.c:6187:... · d2f8b7d0
      Michael Widenius authored
      Fix for LP#602604: RQG: ma_blockrec.c:6187: _ma_apply_redo_insert_row_head_or_tail: Assertion `0' failed on Maria engine recovery
      More DBUG_PRINT (to simplify future debugging)
      Aria: Added STATE_IN_REPAIR, which is set on start of repair. This allows us to see if 'crashed' flag was set intentionally.
      Aria: Some trivial speedup optimization
      Aria: Better warning if table was marked crashed by unfinnished repair
      
      
      mysql-test/lib/v1/mysql-test-run.pl:
        Fix so one can run RQG
      mysql-test/suite/maria/r/maria-recovery2.result:
        Update for new error message.
      mysys/stacktrace.c:
        Fixed compiler warning
      storage/maria/ha_maria.cc:
        More DBUG_PRINT
        Added STATE_IN_REPAIR flag, which is set on start of repair. This allows us to see if 'crashed' flag was set intentionally.
        Don't log query for dropping temporary table.
      storage/maria/ha_maria.h:
        Added prototype for drop_table()
      storage/maria/ma_blockrec.c:
        More DBUG_PRINT
        Make read_long_data() inline for most cases. (Trivial speedup optimization)
      storage/maria/ma_check.c:
        Better warning if table was marked crashed by unfinnished repair
      storage/maria/ma_open.c:
        More DBUG_PRINT
      storage/maria/ma_recovery.c:
        Give warning if found crashed table.
        Changed warning for tables that can't be opened.
      storage/maria/ma_recovery_util.c:
        Write warnings to DBUG file
      storage/maria/maria_chk.c:
        Added STATE_IN_REPAIR flag, which is set on start of repair. This allows us to see if 'crashed' flag was set intentionally.
      storage/maria/maria_def.h:
        Added maria_mark_in_repair(x)
      storage/maria/maria_read_log.c:
        Added option: --character-sets-dir
      storage/maria/trnman.c:
        By default set min_read_from to max value.
        This allows us to remove TRN:s from rows during recovery to get more space.
        This fixes bug LP#602604: RQG: ma_blockrec.c:6187: _ma_apply_redo_insert_row_head_or_tail: Assertion `0' failed on Maria engine recovery
      d2f8b7d0
  15. 27 Dec, 2008 1 commit
    • Michael Widenius's avatar
      Fixed bugs found by pushbuild · 910284e6
      Michael Widenius authored
      Added code to detect and give error when doing an insert into a view where we accessed fields in a not yet read table
      Disabled test in subselect.test as the CHECK_OPTION for views doesn't work for insert.
      This needs to be fixed properly later.
      
      The problem with views are described in Bug #41760 Inserting into multiple-table views is not working
      
      mysql-test/r/insert.result:
        Fixed wrong usage of insert into view.
      mysql-test/r/subselect.result:
        Disabled wrong test (temporary)
      mysql-test/suite/maria/r/maria.result:
        Added test of size of table
      mysql-test/suite/maria/t/maria.test:
        Added test of size of table
      mysql-test/t/insert.test:
        Fixed wrong usage of insert into view
        The bug is that during insert/update we currently don't read any of the referenced tables of the view.
        This means that we can't get a value from another table to use as part of the update.
      mysql-test/t/subselect.test:
        Disabled not working test until someone has time to fix insert into view properly
        Here we where refering to last used value in t2, which is wrong.
      sql/sql_insert.cc:
        Detect if we are trying to update one table in a view based on value in another, not yet read, table.
        This fixes the problem discovered in insert.test
      storage/maria/ma_blockrec.c:
        Don't ignore not critical changes to the last page in the table.
        We need to write the last page as otherwise we can during aborting of a row with a duplicate key get
        state.data_file_length and the real length of file out of sync
      storage/maria/ma_check.c:
        Flush the page cache even if we got an error during zerofill.
        (This fixes a call to assert() in case of a too short data file)
      storage/maria/ma_pagecache.c:
        Mark page as read when we do a write of a full page.
        This fixes a bug when we got an error during read and then used direct write to page to update it
      storage/maria/ma_state.c:
        Restore info->lock.type after call to maria_versioning.
        Fixed crash in maria_recover.test
      storage/maria/maria_read_log.c:
        Don't write thread id in debug log. (Not needed as maria_read_log is a single treaded program)
      910284e6
  16. 24 Apr, 2008 1 commit
    • unknown's avatar
      WL#3072 - Maria Recovery · c9a82581
      unknown authored
      Recovery of R-tree and fulltext indices.
      Fix for BUG#35551 "Maria: crash in REPAIR TABLE/ENABLE KEYS if using
      repair-with-keycache method".
      Fix for bug (see ma_rt_index.c) where we could have a wrong
      page_link pointer causing wrong memory access during some R-tree
      index insert/delete.
      Making ma_rt_test work again (it had been neglected over time) and
      adding options (record type etc) to prepare it for integration into
      ma_test_all-t (but there is BUG#36321 about "ma_rt_test -M" crash)
      
      
      mysql-test/r/maria.result:
        correct result
      mysql-test/t/maria.test:
        now we get no error
      storage/maria/ma_blockrec.c:
        delete_dir_entry() and delete_head_or_tail() don't use info->keyread_buff.
        ma_get_length() does not change **packet, marking it with 'const' to
        remove some casts in callers of this function. The
        (const uchar**)&header casts will be removed when Monty changes 'header'
        to const uchar*.
        _ma_apply_redo_purge_row_head_or_tail() sets 'buff' from pagecache_read()
        so its initialization was superfluous.
      storage/maria/ma_check.c:
        Fix for BUG#35551 "Maria: crash in REPAIR TABLE/ENABLE KEYS if using repair-with-keycache method"
        (see comment in code)
      storage/maria/ma_create.c:
        FULLTEXT and SPATIAL indices have logging now, they are recoverable.
      storage/maria/ma_delete.c:
        Logging done by _ma_ck_delete() is moved to a function
        (_ma_write_undo_key_delete()), for reusal by R-tree logging.
        _ma_log_delete() is made non-static for same
        reason, and some of its parameters are made pointers to const.
        Removed wrong comment ("Note that for delete key" etc, contradicted by
        code and comment "Log also position to row" a few lines above)
      storage/maria/ma_ft_update.c:
        unneeded cast, comment for future
      storage/maria/ma_key_recover.c:
        Comment about possible deadlock.
        Write bad page to DBUG trace if KEY_OP_CHECK founds bad CRC.
        Support operation KEY_OP_MULTI_COPY.
        When we execute, in UNDO phase, UNDO_KEY_DELETE|INSERT, we must call
        the proper key insertion|deletion depending on if this is R-tree
        or B-tree.
        Explanation of of _ma_[un]lock_key_del() work, maybe useful for
        mortals like me.
      storage/maria/ma_key_recover.h:
        change of prototypes
      storage/maria/ma_loghandler.h:
        New operation which can be stored in REDO_INDEX log records: KEY_OP_MULTI_COPY
      storage/maria/ma_page.c:
        Comments
      storage/maria/ma_pagecache.c:
        typo
      storage/maria/ma_rt_index.c:
        Fix for bug: the page_link pointer in maria_rtree_insert_req()
        could be wrong when we set its 'changed' member; for the solution
        see ma_key_recover.h. It is needed only in cases when we manipulate
        several pages.
        Logging of changes done to pages by key insert/delete.
        maria_rtree_delete()'s main work is moved to a new function
        maria_rtree_real_delete(), which is used by maria_rtree_delete()
        and by applying of UNDO_KEY_INSERT.
      storage/maria/ma_rt_index.h:
        new prototypes and macros for ma_rt_index.c
      storage/maria/ma_rt_key.c:
        Logging of maria_rtree_add_key() and maria_rtree_delete_key().
        When inserting, split is necessary if there is not enough room for key:
        take checksum's occupied space in this calculation.
      storage/maria/ma_rt_key.h:
        new prototypes (those functions need to know the page's id
        because they do logging)
      storage/maria/ma_rt_mbr.c:
        Comments about what the functions change.
      storage/maria/ma_rt_split.c:
        maria_rtree_split_page() needs to know the page's id, because
        it does logging.
        Logging of what a split operation does to the split page (see
        comment of _ma_log_rt_split(): moves of keys inside the page,
        sometimes insertion of the new key, and shrinking of the page)
        and to the new page (receives some keys from split page, and
        sometimes the new key).
      storage/maria/ma_rt_test.c:
        ma_rt_test had been forgotten when maria_rkey() was changed some months ago
        (0->HA_WHOLE_KEY change), and when calls to maria_rnd(,,HA_OFFSET_ERROR)
        were rewritten to maria_scan() calls (which implies maria_scan_init()).
        The 'max_i' change is to adapt to the fact that maria_scan() does
        not return deleted records for BLOCK_RECORD but does so for other formats;
        the initial code assumed a certain number of deleted records would be
        returned, we change it to rather count only non-deleted ones.
        We also add more features to this test, like ma_test1 (the plan
        is to run ma_rt_test in ma_test_all-t):
        options to choose records' format, table checksum, transactions,
        checkpoints, end at specific stages, abort without committing,
        and debug trace.
      storage/maria/ma_test1.c:
        MY_INIT() does my_init().
      storage/maria/ma_write.c:
        Logging done by _ma_ck_write_btree_with_log() is moved to a function
        (_ma_write_undo_key_insert()), for reusal by R-tree logging.
        _ma_log_new() and _ma_log_change() are made non-static for same
        reason. Some parameters of logging functions are made pointers to const.
        If EXTRA_DEBUG_KEY_CHANGES, we now log CRC in _ma_log_change() too
        (better checks, bigger record).
      storage/maria/maria_read_log.c:
        Program takes no arguments, bail out if any, instead of silently discarding them
      storage/myisam/rt_test.c:
        rt_test had been forgotten when mi_rkey() was changed some months ago
        (0->HA_WHOLE_KEY change).
        The 'max_i' change is to make it symmetric with ma_rt_test.c
      mysql-test/r/maria-gis-rtree-dynamic.result:
        correct result
      mysql-test/r/maria-gis-rtree-trans.result:
        correct result
      mysql-test/r/maria-recovery-rtree-ft.result:
        almost correct result (hitting BUG# in the end)
      mysql-test/t/maria-gis-rtree-dynamic.test:
        test R-tree & dynamic row format
      mysql-test/t/maria-gis-rtree-trans.test:
        Test R-tree and page row format and transactional
      mysql-test/t/maria-recovery-rtree-ft-master.opt:
        usual options for recovery testing
      mysql-test/t/maria-recovery-rtree-ft.test:
        test of recovery of R-tree and fulltext indices.
      c9a82581
  17. 10 Apr, 2008 1 commit
    • unknown's avatar
      Added versioning of row data · 126c1228
      unknown authored
      Will in future changeset (soon) av versioning of status variables (number of rows) and index
      Changed some LEX_STRING to LEX_CUSTRING to avoid casts and warnings
      Removed some not needed variables (as noticed by Guilhem)
      
      
      include/maria.h:
        Added prototypes for maria_chk_init_for_check(), maria_versioning() and maria_ignore_trids()
      include/my_base.h:
        Add new error HA_ERR_ROW_NOT_VISIBLE
      include/myisamchk.h:
        Added variables for checking visibility of rows during maria_chk
      include/thr_lock.h:
        Changed argument type from int to my_bool for get_status
        Added variable allow_multiple_concurrent_insert, to signal if table supports multiple concurrent inserts
      mysql-test/r/maria-page-checksum.result:
        Added missing drop table
      mysql-test/t/maria-page-checksum.test:
        Added missing drop table
      mysys/my_handler.c:
        Added new error messages
      mysys/thr_lock.c:
        Added support for multiple concurrent inserts, if table handler supports it
      sql/sql_yacc.yy:
        Added LOCK TABLE table_name WRITE CONCURRENT
        This was added (temporarly?) to be able to check versioning with Maria
      storage/csv/ha_tina.cc:
        Updated parameter for get_status
      storage/maria/ha_maria.cc:
        Added calls to maria_chk_init_status()
        Fixed call to ma_control_file_open()
      storage/maria/ma_blockrec.c:
        Changed some LEX_STRING to LEX_CUSTRING to avoid casts and warnings
        Changed back some 'header' parameters to const char*
        Removed some casts
        
        Added support for versioning:
        - If info->row_flag & ROW_FLAG_TRANSID is set, store transaction id together with the row
        - When reading rows, check if rows are visible. Give error if not
        - When scanning table, ignore not visible rows
        - Added function parameters to some functions, to be able to call _ma_compact_block_page() with different parameters depending of if the page is a HEAD or TAIL page
        - _ma_compact_block_page() deletes transaction id's that are visible by all running transactions
        - Added functions for thr_lock() to enable multiple concurrent inserts
        - Added helper function 'mysql_versioning()' to enable/disable versioning
        - Added helper function maria_ignore_trids(), used by maria_chk and maria_pack to see all rows.
      storage/maria/ma_blockrec.h:
        Updated parameters for some functions.
        Added new functions to read/store state with thr_lock
      storage/maria/ma_check.c:
        Enable handling of transaction id's in rows
        Give a readable error if a table contains a transation id that makes rows not visible
      storage/maria/ma_control_file.c:
        Added option to not give warning if control file doesn't exists.
      storage/maria/ma_control_file.h:
        Updated parameter lists for ma_control_file_open()
      storage/maria/ma_delete.c:
        Removed not used variable (suggestion by Guilhem)
      storage/maria/ma_locking.c:
        Changed type of argument from int -> my_bool
      storage/maria/ma_open.c:
        Removed not used variables 'key_write_undo_lsn' and 'key_delete_undo_lsn'
        Added new thr_lock interface functions for BLOCK_RECORD to enable multiple concurrent insert
      storage/maria/ma_test1.c:
        Added option --versioning (-C) to check versioning
      storage/maria/ma_test2.c:
        Added option -C to check versioning
      storage/maria/ma_test_recovery:
        Forward argumetns to ma_test_recovery.pl
      storage/maria/ma_write.c:
        Removed not used variable key_write_undo_lsn
      storage/maria/maria_chk.c:
        Always read control file (if exist) at start
        Initialize checking of tables by calling maria_chk_init_for_check()
        In verbose mode and in case of error, print max found transaction id
      storage/maria/maria_def.h:
        Added Trid to MARIA_ROW to be able to check transaction id for found row
        Moved 'base_length' from MARIA_ROW to MARIA_HA to be able to handle different base length (with and without TRANSID) without if's
        Added default row_flag to MARIA_HA for the same reason
        Changed LEX_STRING -> LEX_CUSTRING to avoid casts in ma_blockrec.c
        Removed not needed variables key_write_undo_lsn and key_delete_undo_lsn
        Added prototypes for new functions and fixed those that had changed
      storage/maria/maria_pack.c:
        Ensure we can read all rows from the file, independent of the used transaction id
      storage/maria/maria_read_log.c:
        Updated arguments to ma_control_file_open()
      storage/maria/trnman.c:
        If we have only one transaction, fixed that min_read_from contains current transaction
        Fixed that trnman_can_read_from() returns that row is readable if it was written by current transaction
      storage/maria/unittest/ma_control_file-t.c:
        Updated arguments to ma_control_file_open()
      storage/maria/unittest/ma_test_all-t:
        Added test of versioning
        Removed printing of one extra space
      storage/maria/unittest/ma_test_loghandler-t.c:
        Updated arguments to ma_control_file_open()
      storage/maria/unittest/ma_test_loghandler_first_lsn-t.c:
        Updated arguments to ma_control_file_open()
      storage/maria/unittest/ma_test_loghandler_max_lsn-t.c:
        Updated arguments to ma_control_file_open()
      storage/maria/unittest/ma_test_loghandler_multigroup-t.c:
        Updated arguments to ma_control_file_open()
      storage/maria/unittest/ma_test_loghandler_multithread-t.c:
        Updated arguments to ma_control_file_open()
      storage/maria/unittest/ma_test_loghandler_noflush-t.c:
        Updated arguments to ma_control_file_open()
      storage/maria/unittest/ma_test_loghandler_nologs-t.c:
        Updated arguments to ma_control_file_open()
      storage/maria/unittest/ma_test_loghandler_pagecache-t.c:
        Updated arguments to ma_control_file_open()
      storage/maria/unittest/ma_test_loghandler_purge-t.c:
        Updated arguments to ma_control_file_open()
      storage/maria/unittest/ma_test_recovery.expected:
        Updated file with result from new tests
      storage/maria/unittest/ma_test_recovery.pl:
        Added options --abort-on-error and --verbose
        In case of --verbose, print all excuted shell commands
        Added test of versioning
      storage/myisam/mi_locking.c:
        Updated type of parameter
      storage/myisam/myisamdef.h:
        Updated type of parameter
      mysql-test/r/maria-mvcc.result:
        New BitKeeper file ``mysql-test/r/maria-mvcc.result''
      mysql-test/t/maria-mvcc.test:
        New BitKeeper file ``mysql-test/t/maria-mvcc.test''
      126c1228
  18. 04 Apr, 2008 1 commit
    • unknown's avatar
      Store maximum transaction id into control file at clean shutdown. · 722a8ebe
      unknown authored
      This can serve to maria_chk to check that trids found in rows and keys
      are not too big. Also used by Recovery when logs are lost.
      Options --require-control-file, --datadir, --log-dir (yes, the dashes are
      inconsistent but I imitated mysqld --datadir and --maria-log-dir) for
      maria_chk.
      Lock control file _before_ reading its content.
      
      
      storage/maria/ha_maria.cc:
        new prototype
      storage/maria/ma_check.c:
        A function to find the max trid in the system (consults transaction
        manager and control file), to check tables.
      storage/maria/ma_checkpoint.c:
        new prototype
      storage/maria/ma_control_file.c:
        Store max trid into control file, in a backward-compatible way
        (can still read old control files).
        Parameter to ma_control_file_open(), to not create the log if it's
        missing (maria_chk needs that).
        Lock control file _before_ reading its content.
        Fix for a segfault when reading an old control file (bzero() with a
        negative second argument)
      storage/maria/ma_control_file.h:
        changes to the control file module's API
      storage/maria/ma_init.c:
        When Maria shuts down cleanly, store max trid into control file.
      storage/maria/ma_loghandler.c:
        new prototype
      storage/maria/ma_recovery.c:
        During recovery, consult max trid stored in control file, in case it is
        bigger than what we found in log (case of logs manually removed by user).
      storage/maria/ma_test1.c:
        new prototype
      storage/maria/ma_test2.c:
        new prototype
      storage/maria/maria_chk.c:
        New option --require-control-file (abort if control file not found),
        --datadir (path for control file (and for logs if --log-dir not specified)),
        --log-dir (path for logs).
        Try to open control file when maria_chk starts.
      storage/maria/maria_read_log.c:
        new prototype
      storage/maria/trnman.c:
        A new function to know max trid in transaction manager
      storage/maria/trnman_public.h:
        New function
      storage/maria/unittest/ma_control_file-t.c:
        new prototypes. Testing storing and retrieving the max trid to/from
        control file
      storage/maria/unittest/ma_test_loghandler-t.c:
        new prototype
      storage/maria/unittest/ma_test_loghandler_first_lsn-t.c:
        new prototype
      storage/maria/unittest/ma_test_loghandler_max_lsn-t.c:
        new prototype
      storage/maria/unittest/ma_test_loghandler_multigroup-t.c:
        new prototype
      storage/maria/unittest/ma_test_loghandler_multithread-t.c:
        new prototype
      storage/maria/unittest/ma_test_loghandler_noflush-t.c:
        new prototype
      storage/maria/unittest/ma_test_loghandler_nologs-t.c:
        new prototype
      storage/maria/unittest/ma_test_loghandler_pagecache-t.c:
        new prototype
      storage/maria/unittest/ma_test_loghandler_purge-t.c:
        new prototype
      722a8ebe
  19. 26 Jan, 2008 1 commit
    • unknown's avatar
      Support of moving logs in other directory then data root. · a1255fc2
      unknown authored
      Fixed typo in maria_read_log "help" output.
      Compilation warning on Mac OS fixed.
      
      
      mysql-test/include/maria_empty_logs.inc:
        Added support of moving maria log relatively
          to master data directory.
      mysql-test/include/maria_make_snapshot_for_feeding_recovery.inc:
        Added support of moving maria log relatively
          to master data directory.
      mysql-test/include/maria_verify_recovery.inc:
        Added support of moving maria log relatively
          to master data directory.
      mysql-test/t/maria-recovery-master.opt:
        Check of recovery with logs in other place.
      mysql-test/t/maria-recovery.test:
        Check of recovery with logs in other place.
      storage/maria/ha_maria.cc:
        Support of moving logs in other directory then data root.
      storage/maria/ma_loghandler.c:
        Compilation warning on Mac OS fixed.
      storage/maria/ma_static.c:
        Support of moving logs in other directory then data root.
      storage/maria/ma_test1.c:
        Support of moving logs in other directory then data root.
      storage/maria/maria_def.h:
        Support of moving logs in other directory then data root.
      storage/maria/maria_read_log.c:
        Support of moving logs in other directory then data root.
        Fixed typo in maria_read_log "help" output.
      a1255fc2
  20. 03 Jan, 2008 1 commit
    • unknown's avatar
      Added wrapper for chmod(): my_chmod() · c719e1fd
      unknown authored
      Automaticly disable ma_test_recovery if not compiled with debugging. This fixes that make test works in Maria.
      Fixed wrong merge of ma_init.c from 5.1
      Portability fixes:
      - Use my_chmod() instead of my_chmod()
      - Use my_access() instead of my_stat() to test if file exists
      - Don't test result value of pthread_mutex_lock() and pthread_mutex_unlock() as this is not portable
      - No reason to test if file exists before we delete it
      
      
      
      include/my_sys.h:
        Added my_chmod
      include/mysys_err.h:
        Added error for my_chmod
      mysys/Makefile.am:
        Added my_chmod
      mysys/errors.c:
        Added error for my_chmod
      mysys/my_init.c:
        Syncronize with 5.1 to fix setting of QueryPerformanceFrequency()
      storage/maria/ma_test1.c:
        Changed short option of --skip-delete and --skip-update to be more logical
      storage/maria/ma_test_recovery.expected:
        Updated results after adding more tests
      storage/maria/ma_test_recovery:
        Abort test nicely if we are runnning without debugging
        Added more tests
        Changed temporary file names so that one can run maria_chk on them
        Removed some old comments
      storage/maria/maria_read_log.c:
        Added note if maria_read_log will not be able to create byte-to-byte identical tables compared to normal execution
      storage/maria/unittest/ma_pagecache_consist.c:
        Removed wrong setting of buff that caused memory overwrite
        Use my_chmod() instead of chmod()
        Don't test result value of pthread_mutex_lock() and pthread_mutex_unlock() as this is not portable
      storage/maria/unittest/ma_pagecache_single.c:
        Use my_chmod() instead of chmod()
        Don't test result value of pthread_mutex_lock() and pthread_mutex_unlock() as this is not portable
      storage/maria/unittest/ma_test_loghandler_first_lsn-t.c:
        No reason to test if file exists before we delete it
      storage/maria/unittest/ma_test_loghandler_multithread-t.c:
        Don't test result value of pthread_mutex_lock() and pthread_mutex_unlock() as this is not portable
      storage/maria/unittest/ma_test_loghandler_noflush-t.c:
        No reason to test if file exists before we delete it
      storage/maria/unittest/ma_test_loghandler_nologs-t.c:
        Use my_access() instead of my_stat() to test if file exists
      storage/maria/unittest/ma_test_loghandler_pagecache-t.c:
        No reason to test if file exists before we delete it
        chmod -> my_chmod
      mysys/my_chmod.c:
        Added wrapper for chmod()
      c719e1fd
  21. 15 Dec, 2007 2 commits
    • unknown's avatar
      Postmerge fix. · fabdf6df
      unknown authored
      storage/maria/ma_loghandler.c:
        Postmerge fix.
        New state of the log added to proicess correctly shutdown.
      storage/maria/ma_loghandler.h:
        New state of the log added to proicess correctly shutdown.
      fabdf6df
    • unknown's avatar
      Transaction log behaviour in case of write · d225521c
      unknown authored
        error fixed (switching to the read only mode).
      Added read only mode of transactions log handler.
      
      
      storage/maria/ha_maria.cc:
        Transaction log initialization parameters change.
      storage/maria/ma_check.c:
        New status variable of transactional log.
      storage/maria/ma_create.c:
        New status variable of transactional log.
      storage/maria/ma_loghandler.c:
        - New status variable added.
        - Checking the status variable in the loghandler
        interface functions added
        - All fails of loghandler functions revised.
        - UNRECOVERABLE_ERROR() removed.
        - Switching to read only mode added where it nead.
        - Checking of log state added before writes log content
          and changing status variables like sent_to_file, n_buffers_only, flushed.
        - Readonly loghandler initialization added.
        - Fixed problem with example table transactional log initialization.
      storage/maria/ma_loghandler.h:
        Readonly loghandler initialization added.
        Fixed problem with example table transactional log initialization.
        New status variable added.
      storage/maria/ma_open.c:
        New status variable of transactional log.
      storage/maria/ma_test1.c:
        Transaction log initialization parameters change.
      storage/maria/ma_test2.c:
        Transaction log initialization parameters change.
      storage/maria/maria_read_log.c:
        Transaction loghandler initialization in read only mode in
        case of only dysplay parameter.
      storage/maria/unittest/Makefile.am:
        Test of readonly mode added.
      storage/maria/unittest/ma_test_loghandler-t.c:
        Transaction log initialization parameters change.
        Fixed problem with example table transactional log initialization.
        Fixed incorrect fprintf call parameters.
      storage/maria/unittest/ma_test_loghandler_first_lsn-t.c:
        Transaction log initialization parameters change.
        Fixed problem with example table transactional log initialization.
      storage/maria/unittest/ma_test_loghandler_max_lsn-t.c:
        Transaction log initialization parameters change.
        Fixed problem with example table transactional log initialization.
      storage/maria/unittest/ma_test_loghandler_multigroup-t.c:
        Useing this test also as read only loghandler test.
        Transaction log initialization parameters change.
        Fixed problem with example table transactional log initialization.
      storage/maria/unittest/ma_test_loghandler_multithread-t.c:
        Transaction log initialization parameters change.
        Fixed problem with example table transactional log initialization.
      storage/maria/unittest/ma_test_loghandler_noflush-t.c:
        Transaction log initialization parameters change.
        Fixed problem with example table transactional log initialization.
      storage/maria/unittest/ma_test_loghandler_pagecache-t.c:
        Transaction log initialization parameters change.
        Fixed problem with example table transactional log initialization.
      storage/maria/unittest/ma_test_loghandler_purge-t.c:
        Transaction log initialization parameters change.
        Fixed problem with example table transactional log initialization.
      d225521c
  22. 12 Dec, 2007 1 commit
    • unknown's avatar
      WL#3072 - Maria Recovery: recovery of state.auto_increment. · f7df9add
      unknown authored
      When we log UNDO_KEY_INSERT for an auto_inc key, we update
      state.auto_increment (not anymore at the end of maria_write() except
      if this is a non-transactional table). When Recovery sees UNDO_KEY_INSERT
      in the REDO phase, it reads the auto_inc value from it and updates
      state.auto_increment.
      
      
      mysql-test/r/maria-recovery.result:
        Without the code fix, there would be in CHECK TABLE:
        "Auto-increment value: 0 is smaller than max used value: 3"
        and no AUTO_INCREMENT= clause in SHOW CREATE TABLE.
      mysql-test/t/maria-recovery.test:
        Test of recovery of state.auto_increment: from an old table,
        does the replaying of the log set state.auto_increment to
        what it should be.
      storage/maria/ma_check.c:
        new way of calling ma_retrieve_auto_increment(): pass key
      storage/maria/ma_key.c:
        ma_retrieve_auto_increment() now operates directly with a pointer
        to the key and not on the record.
      storage/maria/ma_key_recover.c:
        dedicated write_hook_for_undo_key_insert(): sets state.auto_increment
        under log's mutex.
      storage/maria/ma_key_recover.h:
        Dedicated hook for UNDO_KEY_INSERT, to set state.auto_increment.
        Such hook needs a new member st_msg_write_hook_for_undo_key::auto_increment,
        which contains the auto_increment value inserted.
      storage/maria/ma_loghandler.c:
        UNDO_KEY_INSERT gets a dedicated write_hook, to set auto_increment.
      storage/maria/ma_recovery.c:
        When in the REDO phase we see UNDO_KEY_INSERT: if the state is older
        than this record, and the key is the auto_increment one, read
        the key's value from the log record and update state.auto_increment.
      storage/maria/ma_test_all.sh:
        use $maria_path to be able to run from /dev/shm (faster)
      storage/maria/ma_update.c:
        bool is more of C++, using my_bool.
        If table is transactional, state.auto_increment is already updated
        in write_hook_for_undo_key_insert().
      storage/maria/ma_write.c:
        If table is transactional, state.auto_increment is not updated at
        the end of maria_write() but rather in write_hook_for_undo_key_insert()
        (under log's mutex, so that a concurrent checkpoint does not read
        state.auto_increment while it is changing - corrupted).
        _ma_ck_write_btree_with_log() extracts the auto_increment value
        from the key, puts it into msg.auto_increment, and this is passed
        to write_hook_for_undo_key_insert().
      storage/maria/maria_def.h:
        change of prototype of ma_retrieve_auto_increment()
      storage/maria/maria_read_log.c:
        use default log file size. Use separate page caches for table
        and logs (needed if maria_block_size!=TRANSLOG_PAGE_SIZE).
      f7df9add
  23. 04 Dec, 2007 1 commit
    • unknown's avatar
      Added error HA_ERR_FILE_TOO_SHORT to be used when files are shorter than... · ebf7ab7b
      unknown authored
      Added error HA_ERR_FILE_TOO_SHORT to be used when files are shorter than expected (by my_read/my_pread)
      Added debugger hook _my_dbug_put_break_here() that is called if we get a CRC that matches --debug-crc-break (my_crc_dbug_break)
      Fixed REDO_REPAIR to use all repair modes (repair, repair_by_sort, repair_paralell
      REDO_REPAIR now also logs used key map
      Fixed some bugs in REDO logging of key pages
      Better error messages from maria_read_log
      Added my_readwrite_flags to init_pagecache() to be able to get better error messages and simplify code.
      Don't allow pagecaches with less than 8 blocks (Causes strange crashes)
      Added EXTRA_DEBUG_KEY_CHANGES. When this is defined some REDO_INDEX entries contains page checksums (these are calculated and checked in DBUG mode, ignored otherwise)
      Fixed bug in ma_pagecache unit tests that caused program to sometimes fail
      Added some missing calls to MY_INIT() that caused some unit tests to fail
      Fixed that TRUNCATE works properly on temporary MyISAM files
      Updates some result files to new table checksums results (checksum when NULL fields are ignored)
      perl test-insert can be replayed with maria_read_log!
      
      
      sql/share/Makefile.am:
        Change mode to -rw-rw-r--
      BitKeeper/etc/ignore:
        added storage/maria/unittest/page_cache_test_file_1 storage/maria/unittest/pagecache_debug.log
      include/maria.h:
        Added maria_tmpdir
      include/my_base.h:
        Added error HA_ERR_FILE_TOO_SHORT
      include/my_sys.h:
        Added variable my_crc_dbug_check
        Added function my_dbug_put_break_here()
      include/myisamchk.h:
        Added org_key_map (Needed for writing REDO record for REPAIR)
      mysql-test/r/innodb.result:
        Updated to new checksum algorithm (NULL ignored)
      mysql-test/r/mix2_myisam.result:
        Updated to new checksum algorithm (NULL ignored)
      mysql-test/r/myisam.result:
        Updated to new checksum algorithm (NULL ignored)
      mysql-test/t/myisam.test:
        Added used table
      mysys/checksum.c:
        Added DBUG for checksum results
        Added debugger hook so that _my_dbug_put_break_here() is called if we get matching CRC
      mysys/lf_alloc-pin.c:
        Fixed compiler warning
      mysys/my_handler.c:
        Added new error message
      mysys/my_init.c:
        If my_progname is not given, use 'unknown' form my_progname_short
        Added debugger function my_debug_put_break_here()
      mysys/my_pread.c:
        In case of too short file when MY_NABP or MY_FNABP is specified, give error HA_ERR_FILE_TO_SHORT
      mysys/my_read.c:
        In case of too short file when MY_NABP or MY_FNABP is specified, give error HA_ERR_FILE_TO_SHORT
      sql/mysqld.cc:
        Added debug option --debug-crc-break
      sql/sql_parse.cc:
        Trivial optimization
      storage/maria/ha_maria.cc:
        Renamed variable to be more logical
        Ensure that param.testflag is correct when calling repair
        Added extra argument to init_pagecache
        Set default value for maria_tempdir
      storage/maria/ma_blockrec.c:
        Test for HA_ERR_FILE_TOO_SHORT instead for -1
      storage/maria/ma_cache.c:
        Test for HA_ERR_FILE_TOO_SHORT instead for -1
      storage/maria/ma_check.c:
        Set param->testflag to match how repair is run (needed for REDO logging)
        Simple optimization
        Moved flag if page is node from pagelength to keypage-flag byte
        Log used key map in REDO log.
      storage/maria/ma_delete.c:
        Remember previous UNDO entry when writing undo (for future CLR records)
        Moved flag if page is node from pagelength to keypage-flag byte
        Fixed some bugs in redo logging
        Added CRC for some translog REDO_INDEX entries
      storage/maria/ma_dynrec.c:
        Test for HA_ERR_FILE_TOO_SHORT instead for -1
      storage/maria/ma_ft_update.c:
        Fixed call to _ma_store_page_used()
      storage/maria/ma_key_recover.c:
        Added CRC for some translog REDO_INDEX entries
        Removed not needed pagecache_write() in _ma_apply_redo_index()
      storage/maria/ma_locking.c:
        Test for HA_ERR_FILE_TOO_SHORT instead for -1
      storage/maria/ma_loghandler.c:
        Added used key map to REDO_REPAIR_TABLE
      storage/maria/ma_loghandler.h:
        Added operation for checksum of key pages
      storage/maria/ma_open.c:
        Allocate storage for undo lsn pointers
      storage/maria/ma_pagecache.c:
        Remove not needed include file
        Change logging to use fd: for file descritors as other code
        Added my_readwrite_flags to init_pagecache() to be able to get better error messages for maria_chk/maria_read_log
        Don't allow pagecaches with less than 8 blocks
        Remove wrong DBUG_ASSERT()
      storage/maria/ma_pagecache.h:
        Added readwrite_flags
      storage/maria/ma_recovery.c:
        Better error messages for maria_read_log:
        - Added eprint() for printing error messages
        - Print extra \n before error message if we are printing %0 %10 ...
        
        Added used key_map to REDO_REPAIR log entry
        More DBUG
        Call same repair method that was used by mysqld
      storage/maria/ma_rt_index.c:
        Moved flag if page is node from pagelength to keypage-flag byte
      storage/maria/ma_rt_key.c:
        Fixed call to _ma_store_page_used()
      storage/maria/ma_rt_split.c:
        Moved flag if page is node from pagelength to keypage-flag byte
      storage/maria/ma_static.c:
        Added maria_tmpdir
      storage/maria/ma_test1.c:
        Updated call to init_pagecache()
      storage/maria/ma_test2.c:
        Updated call to init_pagecache()
      storage/maria/ma_test3.c:
        Updated call to init_pagecache()
      storage/maria/ma_write.c:
        Removed #ifdef NOT_YET
        Moved flag if page is node from pagelength to keypage-flag byte
        Fixed bug in  _ma_log_del_prefix()
      storage/maria/maria_chk.c:
        Fixed wrong min limit for page_buffer_size
        Updated call to init_pagecache()
      storage/maria/maria_def.h:
        Added EXTRA_DEBUG_KEY_CHANGES. When this is defined some REDO_INDEX entries contains page checksums
        Moved flag if page is node from pagelength to keypage-flag byte
      storage/maria/maria_ftdump.c:
        Updated call to init_pagecache()
      storage/maria/maria_pack.c:
        Updated call to init_pagecache()
        Reset share->state.create_rename_lsn & share->state.is_of_horizon
      storage/maria/maria_read_log.c:
        Better error messages
        Added --tmpdir option (needed to set temporary directory for REDO_REPAIR)
        Added --start-from-lsn
        Changed option for --display-only to 'd' (wanted to use -o for 'offset')
      storage/maria/unittest/lockman2-t.c:
        Added missing call to MY_INIT()
      storage/maria/unittest/ma_pagecache_consist.c:
        Updated call to init_pagecache()
      storage/maria/unittest/ma_pagecache_single.c:
        Fixed bug that caused program to sometimes fail
        Added some DBUG_ASSERTS()
        Changed some calls to malloc()/free() to my_malloc()/my_free()
        Create extra file to expose original hard-to-find bug
      storage/maria/unittest/ma_test_loghandler-t.c:
        Updated call to init_pagecache()
      storage/maria/unittest/ma_test_loghandler_first_lsn-t.c:
        Updated call to init_pagecache()
      storage/maria/unittest/ma_test_loghandler_max_lsn-t.c:
        Updated call to init_pagecache()
      storage/maria/unittest/ma_test_loghandler_multigroup-t.c:
        Updated call to init_pagecache()
      storage/maria/unittest/ma_test_loghandler_multithread-t.c:
        Updated call to init_pagecache()
      storage/maria/unittest/ma_test_loghandler_noflush-t.c:
        Updated call to init_pagecache()
      storage/maria/unittest/ma_test_loghandler_pagecache-t.c:
        Updated call to init_pagecache()
      storage/maria/unittest/ma_test_loghandler_purge-t.c:
        Updated call to init_pagecache()
      storage/maria/unittest/test_file.c:
        Changed malloc()/free() to my_malloc()/my_free()
        Fixed memory leak
        Changd logic a bit while trying to find bug in reset_file()
      storage/maria/unittest/trnman-t.c:
        Added missing call to MY_INIT()
      storage/myisam/mi_cache.c:
        Test for HA_ERR_FILE_TOO_SHORT instead for -1
      storage/myisam/mi_create.c:
        Removed O_EXCL to get TRUNCATE to work for temporary files
      storage/myisam/mi_dynrec.c:
        Test for HA_ERR_FILE_TOO_SHORT instead for -1
      storage/myisam/mi_locking.c:
        Test for HA_ERR_FILE_TOO_SHORT instead for -1
      mysql-test/r/old-mode.result:
        New BitKeeper file ``mysql-test/r/old-mode.result''
      mysql-test/t/old-mode-master.opt:
        New BitKeeper file ``mysql-test/t/old-mode-master.opt''
      mysql-test/t/old-mode.test:
        New BitKeeper file ``mysql-test/t/old-mode.test''
      ebf7ab7b
  24. 28 Nov, 2007 1 commit
    • unknown's avatar
      Fixed repair_by_sort to work with BLOCK_RECORD · 4e0964cb
      unknown authored
      Fixed bugs in undo logging
      Fixed bug where head block was split before min_row_length (caused Maria to believe row was crashed on read)
      Reserved place for reference-transid on key pages (for packing of transids)
      ALTER TABLE and INSERT ... SELECT now uses fast creation of index
          
      Known bugs:
      ma_test_recovery fails because of a bug in redo handling when log is cut directly after a redo (Guilhem knows how to fix)
      ma_test_recovery.excepted is not totally correct, because of the above bug
      mysqld sometimes fails to restart; Fails with error "end_of_redo_phase: Assertion `long_trid != 0' failed"; Guilhem to investigate
      
      
      include/maria.h:
        Prototype changes
        Added current_filepos to st_maria_sort_info
      mysql-test/r/maria.result:
        Updated results that changes as alter table and insert ... select now uses fast creation of index
      mysys/mf_iocache.c:
        Reset variable to gurard against double invocation
      storage/maria/ma_bitmap.c:
        Added _ma_bitmap_reset_cache() (needed for repair)
      storage/maria/ma_blockrec.c:
        Simplify code
        More initial allocations
        Fixed bug where head block was split before min_row_length (caused Maria to believe row was crashed on read)
      storage/maria/ma_blockrec.h:
        Moved TRANSID_SIZE to maria_def.h
        Added prototype for new functions
      storage/maria/ma_check.c:
        Simplicy code
        Fixed repair_by_sort to work with BLOCK_RECORD
        - When using BLOCK_RECORD or UNPACK create new Maria handle
        - Use common initializer function
        - Align code with maria_repair()
        
        Made some changes to maria_repair_parallel() to use common initializer function
        Removed ASK_MONTY section by fixing noted problem
      storage/maria/ma_close.c:
        Moved check for readonly to _ma_state_info_write()
      storage/maria/ma_key_recover.c:
        Use different log entries if key root changes or not.
        This fixed some bugs when tree grows
      storage/maria/ma_key_recover.h:
        Added keynr to st_msg_to_write_hook_for_undo_key
      storage/maria/ma_loghandler.c:
        Added INIT_LOGREC_UNDO_KEY_INSERT_WITH_ROOT
      storage/maria/ma_loghandler.h:
        Added INIT_LOGREC_UNDO_KEY_INSERT_WITH_ROOT
      storage/maria/ma_open.c:
        Added TRANSID to all key pages (for future compressing of trans id's)
        For compressed records, alloc a bit bigger buffer to avoid valgrind warnings
        If table is opened readonly, don't update state
      storage/maria/ma_packrec.c:
        Allocate bigger array for bit unpacking to avoid valgrind errors
      storage/maria/ma_recovery.c:
        Added UNDO_KEY_INSERT_WITH_ROOT & UNDO_KEY_DELETE_WITH_ROOT
      storage/maria/ma_sort.c:
        More logging
      storage/maria/ma_test_all.sh:
        More tests
      storage/maria/ma_test_recovery.expected:
        Update results
        Note that this is not complete becasue of a bug in recovery
      storage/maria/ma_test_recovery:
        Removed recreation of index (not needed when we have redo for index pages)
      storage/maria/maria_chk.c:
        When using flag --read-only, don't update status for files
        When using --unpack, don't use REPAIR_BY_SORT if other repair option is given
        Enable repair_by_sort for BLOCK records
        Removed not needed newline at start of --describe
      storage/maria/maria_def.h:
        Support for TRANSID_SIZE to key pages
      storage/maria/maria_read_log.c:
        renamed --only-display to --display-only
      4e0964cb
  25. 16 Nov, 2007 1 commit
    • unknown's avatar
      WL#3071 Maria checkpoint, WL#3072 Maria recovery · fc0a25ec
      unknown authored
      instead of fprintf(stderr) when a task (with no user connected) gets
      an error, use my_printf_error(). Flags ME_JUST_WARNING and ME_JUST_INFO
      added to my_error()/my_printf_error(), which pass it to
      my_message_sql() which is modified to call the appropriate
      sql_print_*(). This way recovery can signal its start and end with
      [Note] and not [ERROR] (but failure with [ERROR]).
      Recovery's detailed progress (percents etc) still uses stderr as they
      have to stay on one single line.
      sql_print_error() changed to use my_progname_short (nicer display).
      mysql-test-run.pl --gdb/--ddd does not run mysqld, because
      a breakpoint in mysql_parse is too late to debug startup problems;
      instead, dev should set the breakpoints it wants and then "run" ("r").
      
      
      include/my_sys.h:
        new flags to tell error_handler_hook that this is not an error
        but an information or warning
      mysql-test/mysql-test-run.pl:
        when running with --gdb/--ddd to debug mysqld, breaking at mysql_parse
        is too late to debug startup problems; now, it does not run mysqld,
        does not set breakpoints, developer can set as early breakpoints
        as it wants and is responsible for typing "run" (or "r")
      mysys/my_init.c:
        set my_progname_short
      mysys/my_static.c:
        my_progname_short added
      sql/mysqld.cc:
        * my_message_sql() can now receive info or warning, not only error;
        this allows mysys to tell the user (or the error log if no user)
        about an info or warning. Used from Maria.
        * plugins (or engines like Maria) may want to call my_error(), so
        set up the error handler hook (my_message_sql) before initializing
        plugins; otherwise they get my_message_no_curses which is less
        integrated into mysqld (is just fputs())
        * using my_progname_short instead of my_progname, in my_message_sql()
        (less space on screen)
      storage/maria/ma_checkpoint.c:
        fprintf(stderr) -> ma_message_no_user()
      storage/maria/ma_checkpoint.h:
        function for any Maria task, not connected to a user (example:
        checkpoint, recovery; soon could be deleted records purger)
        to report a message (calls my_printf_error() which, when inside ha_maria,
        leads to sql_print_*(), and when outside, leads to
        my_message_no_curses i.e. stderr).
      storage/maria/ma_recovery.c:
        To tell that recovery starts and ends we use ma_message_no_user()
        (sql_print_*() in practice). Detailed progress info still uses
        stderr as sql_print() cannot put several messages on one line.
        071116 18:42:16 [Note] mysqld: Maria engine: starting recovery
        recovered pages: 0% 67% 100% (0.0 seconds); transactions to roll back: 1 0 (0.0
        seconds); tables to flush: 1 0 (0.0 seconds);
        071116 18:42:16 [Note] mysqld: Maria engine: recovery done
      storage/maria/maria_chk.c:
        my_progname_short moved to mysys
      storage/maria/maria_read_log.c:
        my_progname_short moved to mysys
      storage/myisam/myisamchk.c:
        my_progname_short moved to mysys
      fc0a25ec
  26. 14 Nov, 2007 1 commit
    • unknown's avatar
      WL#3072 - Maria recovery · 1bc5e3b9
      unknown authored
      maria_read_log used to always print a warning message at startup
      to say it is unsafe if ALTER TABLE was used. Now it prints it only
      if the log does show the problem (=ALTER TABLE or CREATE SELECT, which
      both disable logging of REDO_INSERT*).
      For that, when ha_maria::external_lock() disables transactionality
      it writes a LOGREC_INCOMPLETE_LOG to the log, which "maria_read_log -a"
      picks up to write a warning.
      REPAIR TABLE also disables those REDO_INSERT* but as maria_read_log
      executes LOGREC_REDO_REPAIR no warning is needed.
      
      
      storage/maria/ha_maria.cc:
        as we now log a record when disabling transactionility, we need the
        TRN to be set up first
      storage/maria/ma_blockrec.c:
        comment
      storage/maria/ma_loghandler.c:
        new type of log record
      storage/maria/ma_loghandler.h:
        new type of log record
      storage/maria/ma_recovery.c:
        * maria_apply_log() now returns a count of warnings. What currently
        produces warnings is:
        - skipping applying UNDOs though there are some (=> inconsistent table)
        - replaying log (in maria_read_log) though the log contains some
        ALTER TABLE or CREATE SELECT (log misses REDO_INSERT* for those
        and is so incomplete).
        Count of warnings affects the final message of maria_read_log and
        recovery (though in recovery none of the two conditions above should
        happen).
        * maria_read_log used to always print a warning message at startup
        to say it is unsafe if ALTER TABLE was used. Now it prints it only
        if the log does show the problem, i.e. ALTER TABLE or CREATE SELECT
        was used (both disable logging of REDO_INSERT* as those records are
        not needed for recovery; those missing records in turn make
        recreation-from-scratch, via maria_read_log, impossible). For that,
        when ha_maria::external_lock() disables transactionality,
        _ma_tmp_disable_logging_for_table() writes a LOGREC_INCOMPLETE_LOG to
        the log, which maria_apply_log() picks up to write a warning.
      storage/maria/ma_recovery.h:
        maria_apply_log() returns a count of warnings
      storage/maria/maria_def.h:
        _ma_tmp_disable_logging_for_table() grows so becomes a function
      storage/maria/maria_read_log.c:
        maria_apply_log can now return a count of warnings, to temper the
        "SUCCESS" message printed in the end by maria_read_log.
        Advise users to make a backup first.
      1bc5e3b9
  27. 16 Oct, 2007 1 commit
    • unknown's avatar
      WL#3072 Maria recovery · 08b71528
      unknown authored
      - fixes (in recovery, checkpoint, log handler) of bugs found
      during testing.
      - new option --check for maria_read_log: with --only-display (which only
      reads the header) it reads the full record, for debugging.
      
      
      storage/maria/ma_loghandler.c:
        importing patch from Sanja for bug of translog_next_LSN() found
        during recovery
      storage/maria/ma_loghandler_lsn.h:
        better types (0L is 4 bytes on some platforms, it causes problems
        when used into lsn_store(): right shift >= width of type.
      storage/maria/ma_pagecache.c:
        work around infamous "PAGECACHE_PLAIN_PAGE used for transactional
        tables in specialm case"; REDO phase disables logging and this causes
        pages to be PAGECACHE_PLAIN_PAGE, thus ignored wrongly by the
        checkpoint taken at the end of the REDO phase.
      storage/maria/ma_recovery.c:
        - a #ifdef which broke maria_read_log in non-debug builds (no output!)
        - support for maria_read_log --check
        - detect record corruption before opening the table
        - updating is_of_horizon requires writing the state
        - fix for wrong parsing of checkpoint record by recovery
      storage/maria/ma_recovery.h:
        support for maria_read_log --check
      storage/maria/maria_read_log.c:
        Option --check: --only-display only looks at the header;
        adding --check tries a translog_read_record() to see if record can
        be fully read (this is to find bugs).
      08b71528
  28. 08 Oct, 2007 1 commit
    • unknown's avatar
      WL#3072 Maria recovery · 63ff9877
      unknown authored
      Misc changes:
      - fix for benign Valgrind error, compiler warnings
      - fix for a segfault in execution of maria_delete_all_rows() and one
      when taking multiple checkpoints
      - fix for too paranoid assertion
      - adding ability to take checkpoints at the end of the REDO phase
      and at the end of recovery.
      - other minor changes
      
      
      storage/maria/ha_maria.cc:
        The checkpoint done after Recovery is finished, is moved to
        maria_recover().
      storage/maria/ma_bitmap.c:
        fix for Valgrind error: the "shadow debug copy" of the bitmap page
        started unitialized and so ma_print_bitmap() would use it uninitialized
      storage/maria/ma_checkpoint.c:
        * reset pointers to NULL after freeing them, or we segfault at
        next checkpoint in my_realloc().
        * fix for compiler warnings.
      storage/maria/ma_delete_all.c:
        info->trn is NULL for non-transactional tables
      storage/maria/ma_locking.c:
        correct assertion (it fired wrongly in execution of REDO_DROP_TABLE
        due to the maria_extra(HA_PREPARE_FOR_DROP)->_ma_decrement_open_count()
        ->maria_lock_database(F_UNLCK); another solution would have been to
        not call _ma_decrement_open_count() (it's ok to have a wrong open
        count in a table which we are dropping), but the same problem
        would still exist for REDO_RENAME_TABLE.
      storage/maria/ma_loghandler.c:
        fail early if UNRECOVERABLE_ERROR
      storage/maria/ma_recovery.c:
        * new argument to maria_apply_log(): should it take checkpoints
        (at end of REDO phase and at the very end) or no.
        * moving the call to translog_next_LSN() into
        parse_checkpoint_record() ("hide the details").
        * Refining an error detection for something which could happen
        if there is a checkpoint record in the log.
        * Using close_one_table() instead of maria_extra(HA_EXTRA_PREPARE_FOR_DROP|RENAME),
        as it looks safer, and also changing how close_one_table() works:
        it now limits itself to scanning all_tables[], thus having one loopp
        instead of two, which should be faster (as a result, it does not
        close tables not registered in this array, which is ok as there
        should not be any).
      storage/maria/ma_recovery.h:
        new parameter
      storage/maria/maria_read_log.c:
        update to new prototype
      63ff9877
  29. 03 Oct, 2007 1 commit
    • unknown's avatar
      WL#3072 Maria recovery. · 24db7ed7
      unknown authored
      * Thanks to Serg's tip, we fix here the compilation issue of
      REDO_REPAIR_TABLE's execution, by defining versions of
      _ma_killed_ptr() and _ma_check_print_info|warning|error()
      in maria_read_log.c (we move those of maria_chk.c into an include
      file and include it in maria_chk.c and maria_read_log.c).
      Execution of such record looks like working from my tests (it only
      happens in maria_read_log; recovery-from-mysqld skips DDLs and
      REPAIR is considered DDL here as it bypasses logging): tested
      ALTER TABLE ENABLE KEYS and then remove table, apply log: that
      did a repair.
      * Recent changes broke maria_read_log a bit: -a became default
      and -o caused error; fixing this.
      
      
      storage/maria/Makefile.am:
        addind new file
      storage/maria/ma_recovery.c:
        enable execution of REDO_REPAIR_TABLE by maria_read_log now that
        it compiles. Now reason to keep only T_QUICK from testflag.
      storage/maria/maria_chk.c:
        moving these functions to ma_check_standalone.h for reusability
      storage/maria/maria_def.h:
        comment
      storage/maria/maria_read_log.c:
        ma_check_standalone.h needs my_progname_short.
        Fixing bug where "maria_read_log" would default to -a and
        "maria_read_log -o" would throw an error. Implemented behaviour is:
        - no options: usage()
        - -a : applys, applys UNDOs by default unless --disable-undo
        - -o : only prints
      storage/maria/ma_check_standalone.h:
        All standalone programs which need to use functions from ma_check.c
        (like maria_repair()) must define their version of _ma_killed_ptr()
        and _ma_check_print_info|warning|error(). Indeed, linking with ma_check.o
        brings in the dependencies of ma_check.o which are definitions of the above
        functions; if the program does not define them then the ones of
        ha_maria.o are used i.e. ha_maria.o is linked into the program, and this
        brings dependencies of ha_maria.o on mysqld.o into the program's linking
        which thus fails, as the program is not linked with mysqld.o.
        We put in this file the functions which maria_chk.c uses, so that
        they can be reused by maria_read_log (when it replays REDO_REPAIR_TABLE)
        as they are good enough (they just print to stdout/stderr like
        maria_read_log already does).
      24db7ed7
  30. 27 Sep, 2007 1 commit
    • unknown's avatar
      Remove SAFE_MODE for opt_range as it disables UPDATE to use keys · c9c58163
      unknown authored
      REDO optimization (Bascily avoid moving blocks from/to pagecache)
      More command line arguments to maria_read_log
      Fixed recovery bug when recreating table
      
      
      sql/opt_range.cc:
        Remove SAFE_MODE for opt_range as it disables UPDATE to use keys
      storage/maria/ma_blockrec.c:
        REDO optimization
        Use new interface for pagecache_reads to avoid copying page buffers
      storage/maria/ma_loghandler.c:
        Patch from Sanja:
        - Added new parameter to translog_get_page to use direct links to pagecache
        - Changed scanner to be able to use direct links
        
        This avoids a lot of calls to bmove512() in page cache.
      storage/maria/ma_loghandler.h:
        Added direct link to pagecache objects
      storage/maria/ma_open.c:
        Added const to parameter
        Added missing braces
      storage/maria/ma_pagecache.c:
        From Sanja:
        - Added direct links to pagecache (from pagecache_read())
          Dirrect link means that on pagecache_read we get back a pointer to the pagecache buffer
        
        
        From Monty:
        - Fixed arguments to init_page_cache to handle big page caches
        - Fixed compiler warnings
        - Replaced PAGECACHE_PAGE_LINK with PAGECACHE_BLOCK_LINK * to catch errors
      storage/maria/ma_pagecache.h:
        Changed block numbers from int to long to be able to handle big page caches
        Changed some PAGECACHE_PAGE_LINK to PAGECACHE_BLOCK_LINK
      storage/maria/ma_recovery.c:
        Fixed recovery bug when recreating table (table was kept open)
        Moved some variables to function start (portability)
        Added space to some print messages
      storage/maria/maria_chk.c:
        key_buffer_size -> page_buffer_size
      storage/maria/maria_def.h:
        Changed default page_buffer_size to 10M
      storage/maria/maria_read_log.c:
        Added more startup options:
        --version
        --undo (apply undo)
        --page_cache_size (to run with big cache sizes)
        --silent (to not get any output from --apply)
      storage/maria/unittest/ma_control_file-t.c:
        Fixed compiler warning
      storage/maria/unittest/ma_test_loghandler-t.c:
        Added new argument to translog_init_scanner()
      storage/maria/unittest/ma_test_loghandler_multigroup-t.c:
        Added new argument to translog_init_scanner()
      storage/maria/unittest/ma_test_loghandler_multithread-t.c:
        Added new argument to translog_init_scanner()
      c9c58163
  31. 15 Sep, 2007 1 commit
    • unknown's avatar
      WL#3072 Maria Recovery · 9c2ff270
      unknown authored
      * recovery from ha_maria now skips replaying DDLs (too dangerous)
      * maria_read_log still replays DDLs, print warning about issues
      * fixes to replaying of REDO_RENAME
      * don't replay DDLs on corrupted tables (safer)
      * print a one-line message when really doing a recovery (applies to
      ha_maria, not maria_read_log) i.e. some REDOs or UNDOs are read.
      
      
      storage/maria/ma_checkpoint.c:
        fix for assertion failure
      storage/maria/ma_recovery.c:
        * Recovery from ha_maria now skips replaying DDLs (as the initial
        plan said) as this is unsafe in case of crashes during the DDL;
        applying the records may do harm (destroy important files)
        so we prefer to leave the "mess" of files untouched. A proper
        recovery of DDLs requires very careful thinking, probably testing
        separately the existence of the data and index file instead of
        using maria_open() which tests the existence of both, and maybe
        storing create_rename_lsn in the data file too.
        * maria_read_log still replays DDLs, we print a warning about dangers
        (due to ALTER TABLE not logging insertions into the tmp table; we
        will maybe need an option to have logging of those insertions).
        * fixes to replaying of REDO_RENAME (test create_rename_lsn of 'new_name'
        table if it exists; if that table exists and is more recent than the
        record, remove the 'old_name' table).
        * don't replay DDLs on corrupted tables (play safe)
        * fail also in non-debug builds if table is open when it should not be
        (when creating it for example, it should not be already open).
        * when the trace file is not stdout (i.e. when this is ha_maria),
        if really doing a recovery (reading REDOs or UNDOs), print a one-line
        message to stderr to inform about start and end of recovery
        (useful to know what mysqld is doing, especially if it takes long
        or crashes).
      storage/maria/ma_recovery.h:
        parameter to replay DDLs or not
      storage/maria/maria_read_log.c:
        replay DDLs in maria_read_log, to be able to recreate tables from
        scratch.
      9c2ff270
  32. 13 Sep, 2007 1 commit
    • unknown's avatar
      Fixes of the empty log problem. · a303f5b2
      unknown authored
      storage/maria/ma_checkpoint.c:
        The new macro for easier printing LSN added.
      storage/maria/ma_loghandler.c:
        The assertion returned.
        The new macro for easier printing LSN added.
      storage/maria/ma_loghandler_lsn.h:
        The new macro for easier printing LSN added.
      storage/maria/ma_pagecache.c:
        The new macro for easier printing LSN added.
      storage/maria/ma_recovery.c:
        Recovery checks empty log state.
        RECHEADER_READ_ERROR means some real error.
      storage/maria/maria_read_log.c:
        Read log starts from real beggining of the log and precess
        error and empty log states.
        The new macro for easier printing LSN added.
      storage/maria/unittest/ma_test_loghandler-t.c:
        The new macro for easier printing LSN added.
      storage/maria/unittest/ma_test_loghandler_first_lsn-t.c:
        The new macro for easier printing LSN added.
      storage/maria/unittest/ma_test_loghandler_max_lsn-t.c:
        The new macro for easier printing LSN added.
      storage/maria/unittest/ma_test_loghandler_multigroup-t.c:
        The new macro for easier printing LSN added.
      storage/maria/unittest/ma_test_loghandler_multithread-t.c:
        The new macro for easier printing LSN added.
      storage/maria/unittest/ma_test_loghandler_noflush-t.c:
        The new macro for easier printing LSN added.
      a303f5b2
  33. 10 Sep, 2007 1 commit
  34. 09 Sep, 2007 1 commit
    • unknown's avatar
      Added applying of undo for updates · 155193a6
      unknown authored
      Fixed bug in duplicate key handling for block records during repair
      All read-row methods now return error number in case of error
      Don't calculate checksum for null fields
      Fixed bug when running maria_read_log with -o
      
      
      BUILD/SETUP.sh:
        Added STACK_DIRECTION
      BUILD/compile-pentium-debug-max:
        Moved STACK_DIRECTION to SETUP
      include/myisam.h:
        Added extra parameter to write_key
      storage/maria/ma_blockrec.c:
        Added applying of undo for updates
        Fixed indentation
        Removed some not needed casts
        Fixed wrong logging of CLR record
        Split ma_update_block_record to two functions to be able to reuse it from undo-applying
        Simplify filling of packed fields
        ma_record_block_record) now returns error number on failure
        Sligtly changed log record information for undo-update
      storage/maria/ma_check.c:
        Fixed bug in duplicate key handling for block records during repair
      storage/maria/ma_checksum.c:
        Don't calculate checksum for null fields
      storage/maria/ma_dynrec.c:
        _ma_read_dynamic_reocrd() now returns error number on error
        Rest of the changes are code simplification and indentation fixes
      storage/maria/ma_locking.c:
        Added comment
      storage/maria/ma_loghandler.c:
        More debugging
        Removed printing of total_record_length as this was always same as record_length
      storage/maria/ma_open.c:
        Allocate bitmap for changed fields
      storage/maria/ma_packrec.c:
        read_record now returns error number on error
      storage/maria/ma_recovery.c:
        Fixed wrong arguments to undo_row_update
      storage/maria/ma_statrec.c:
        read_record now returns error number on error (not 1)
        Code simplification
      storage/maria/ma_test1.c:
        Added exit possibility after update phase (to test undo of updates)
      storage/maria/maria_def.h:
        Include bitmap header file
      storage/maria/maria_read_log.c:
        Fixed bug when running with -o
      155193a6
  35. 06 Sep, 2007 1 commit
    • unknown's avatar
      WL#3072 Maria Recovery · ac4ad9bd
      unknown authored
      misc fixes of execution of UNDOs in the UNDO phase:
      - into the CLR_END, store the LSN of the _previous_ UNDO (we debated
      what was best, so far we're going with "previous"; later we can change
      to "current" if needed), and store the type of record which is being
      undone (needed to know how to update state.records when we see the
      CLR_END during the REDO phase).
      - declaring all UNDOs and CLR_END as "compressed"
      - when executing an UNDO in the UNDO phase, state.records is updated
      as a hook when writing CLR_END (needed for "recovery of the state"),
      and so is trn->undo_lsn (needed for when we have checkpoints).
      - bugfix (execution of UNDO_ROW_DELETE didn't store the correct checksum
      into the re-inserted row, maria_chk -r thus threw the row away).
      - modifications of ma_test1: where to stop is now driven by --testflag;
      --test-undo just tells how to stop (flush data, flush log, nothing).
      - ma_test_recovery: testing of the UNDO phase, more testing of the
      REDO phase, identification of a bug.
      
      
      storage/maria/ma_blockrec.c:
        - bugfix: execution of UNDO_ROW_DELETE didn't store the correct
        checksum into the row (leading to "maria_chk -r" eliminating the
        re-inserted row, net effect was that rollback appeared to have
        rolled back no deletion). Reason was that write_block_record() used
        info->cur_row.checksum, while "row" can be != &info->cur_row
        (case of UNDO_ROW_DELETE). After fixing this, problems with
        _ma_update_block_record() appeared; indeed checksum was computed
        by  allocate_and_write_block_record() while _ma_update_block_record()
        directly calls write_block_record(). Solution is to compute checksum
        in write_block_record() instead.
        - when executing an UNDO, we now pass the LSN of the _previous_ UNDO
        to block_format functions. This LSN can be 0 (if the being-executed UNDO
        was the transaction's first UNDO), so "undo_lsn==0" cannot work
        anymore to indicate "this is not UNDO work". Using undo_lsn==LSN_ERROR
        instead (this is an impossible LSN).
        - store into CLR_END the type of log record which was undone
        (INSERT/UPDATE/DELETE); needed for Recovery to know if/how it has
        to update state.records if it sees this CLR_END in the REDO phase.
        - when writing the CLR_END in _ma_apply_undo_row_insert(),
        the place to store file's id is log_data+LSN_STORE_SIZE.
        - in _ma_apply_undo_row_insert(), the records-- is moved
        to a hook when writing the CLR_END (this way it is under log's mutex
        which is needed for "recovery of the state")
      storage/maria/ma_loghandler.c:
        - all UNDOs, and CLR_END, start with the LSN of another UNDO; so
        we can declare them "compressed".
        - write_hook_for_clr_end() to set trn->undo_lsn (to the previous
        UNDO's LSN) under log's lock (like UNDOs set trn->undo_lsn under log's
        lock), and also update, if appropriate, state.records.
        - reset share->id to 0 when deassigning; not useful for now but
        sounds logical.
      storage/maria/ma_recovery.c:
        - if no table is found for a REDO, it's not an error; for an UNDO, it is
        - in the REDO phase, when we see a CLR_END we must update trn->undo_lsn
        and sometimes state.records.
        - in the UNDO phase, when we execute an UNDO_ROW_INSERT:
          * update trn->undo_lsn only after executing the record
          * store the _previous_ undo_lsn into the CLR_END
        - at the end of the REDO phase, when we recreate TRN objects, they
        have already their long id in the log (either via a
        LOGREC_LONG_TRANSACTION_ID, or in a checkpoint record), don't write
        a new, useless LOGREC_LONG_TRANSACTION_ID for them.
      storage/maria/ma_test1.c:
        * where to stop execution is now driven by --testflag and not --test-undo
        (ma_test2 already has --testflag for the same purpose). This allows
        us to do a clean stop (with commit) at any point.
        * --test-undo=# tells how to abort (flush all pages (which implies
        flushing log) or only log or nothing); all such "ways of crashing"
        are tested in ma_test_recovery
      storage/maria/ma_test_recovery:
        * Testing execution of UNDOs, with and without BLOBs.
        * Testing idempotency of REDOs.
        * See @todo for a probable bug with BLOBs.
        * maria_chk -rq instead of -r, as with -q it nicely stops on any
        problem in the data file (like the checksum bug see comment of
        ma_blockrec.c).
        * Testing if log was written by UNDO phase (often expected),
        not written by REDO phase (always expected).
        * Less output on the screen, compares with expected output in the end.
        * some shell thingies like "set --" and $# are courtesy of
        Danny and Pekka.
      storage/maria/maria_read_log.c:
        when only displaying the records, don't do an UNDO phase
      storage/maria/ma_test_recovery.expected:
        This is the expected output of a great part of ma_test_recovery.
        ma_test_recovery compares its output to the expected output
        and tells if different.
        If we look at this file it mentions differences in checksum
        (normal, it's not recovered yet) and in records count
        (getting a correct records' count when recovery starts on an
        already existing table, like when testing rollback,
        is coded but not yet pushed).
      ac4ad9bd
  36. 04 Sep, 2007 1 commit
    • unknown's avatar
      Added UNDO handling of insert during recovery · 03437ea0
      unknown authored
      storage/maria/ma_blockrec.c:
        Added UNDO handling of insert during recovery
        To do this, I also had to add write locking of tail pages during undo phase (As we need to access the same page twice if extents are split over two pages)
        Another way to handle the undo of insert would be to store the extent information as part of the UNDO_INSERT block.
      storage/maria/ma_blockrec.h:
        Added new prototype
      storage/maria/ma_loghandler.c:
        Changed type of CLR_END (to avoid crash in log handler)
        Removed not used variable
      storage/maria/ma_loghandler.h:
        Added TRN argument to record_execute_in_undo_phase()
      storage/maria/ma_pagecache.c:
        Hack for undo phase of recovery.  During REDO we work with PLAIN pages, but UNDO works with LSN pages, which caused an abort when trying to access a cached page.
      storage/maria/ma_recovery.c:
        Added execution of UNDO_ROW_INSERT
      storage/maria/ma_test1.c:
        Added option --test-undo for testing recovery with undo
      storage/maria/maria_read_log.c:
        Added processing of undos
      03437ea0
  37. 29 Aug, 2007 2 commits
    • unknown's avatar
      WL#3072 Maria recovery · e27890ca
      unknown authored
      * create page cache before initializing engine and not after, because
      Maria's recovery needs a page cache
      * make the creation of a bitmap page more crash-resistent
      * bugfix (see ma_blockrec.c)
      * back to old way: create an 8k bitmap page when creating table
      * preparations for the UNDO phase: recreate TRNs
      * preparations for Checkpoint: list of dirty pages, testing
      of rec_lsn to know if page should be skipped during Recovery
      (unused in this patch as no Checkpoint module pushed yet)
      * maria_chk tags repaired table with a special LSN
      * reworking all around in ma_recovery.c (less duplication)
      
      
      mysys/my_realloc.c:
        noted an issue in my_realloc()
      sql/mysqld.cc:
        page cache needs to be created before engines are initialized,
        because Maria's initialization may do a recovery which needs
        the page cache.
      storage/maria/ha_maria.cc:
        update to new prototype
      storage/maria/ma_bitmap.c:
        when creating the first bitmap page we used chsize to 8192 bytes then 
        pwrite (overwrite) the last 2 bytes (8191-8192). If crash between
        the two operations, this leaves a bitmap page full without its end
        marker. A later recovery may try to read this page and find it
        exists and misses a marker and conclude it's corrupted and fail.
        Changing the chsize to only 8190 bytes: recovery will then find
        the page is too short and recreate it entirely.
      storage/maria/ma_blockrec.c:
        Fix for a bug: when executing a REDO, if the data page is created,
        data_file_length was increased before _ma_bitmap_set():
        _ma_bitmap_set() called _ma_read_bitmap_page() which, due to the
        increased data_file_length, expected to find a bitmap page on disk
        with a correct end marker; if the bitmap page didn't exist already
        in fact, this failed. Fixed by increasing data_file_length only after
        _ma_read_bitmap_page() has created the new bitmap page correctly.
        This bug could happen every time a REDO is about creating a new
        bitmap page.
      storage/maria/ma_check.c:
        empty data file has a bitmap page
      storage/maria/ma_control_file.c:
        useless parameter to ma_control_file_create_or_open(), just
        test if this is recovery.
      storage/maria/ma_control_file.h:
        new prototype
      storage/maria/ma_create.c:
        Back to how it was before: maria_create() creates an 8k bitmap page.
        Thus (bugfix) data_file_length needs to reflect this instead of being 0.
      storage/maria/ma_loghandler.c:
        as ma_test1 and ma_test2 now use real transactions and not
        dummy_transaction_object, REDO for INSERT/UPDATE/DELETE are always
        about real transactions, can assert this.
        A function for Recovery to assign a short id to a table.
      storage/maria/ma_loghandler.h:
        new function
      storage/maria/ma_loghandler_lsn.h:
        maria_chk tags repaired tables with this LSN
      storage/maria/ma_open.c:
        * enforce that DMLs on transactional tables use real transactions
        and not dummy_transaction_object.
        * test if table was repaired with maria_chk (which has to been
        seen as an import of an external table into the server), test
        validity of create_rename_lsn (header corruption detection)
        * comments.
      storage/maria/ma_recovery.c:
        * preparations for the UNDO phase: recreate TRNs
        * preparations for Checkpoint: list of dirty pages, testing
        of rec_lsn to know if page should be skipped during Recovery
        (unused in this patch as no Checkpoint module pushed yet)
        * reworking all around (less duplication)
      storage/maria/ma_recovery.h:
        a parameter to say if the UNDO phase should be skipped
      storage/maria/maria_chk.c:
        tag repaired tables with a special LSN
      storage/maria/maria_read_log.c:
        * update to new prototype
        * no UNDO phase in maria_read_log for now
      storage/maria/trnman.c:
        * a function for Recovery to create a transaction (TRN), needed
        in the UNDO phase
        * a function for Recovery to grab an existing transaction, needed
        in the UNDO phase (rollback all existing transactions)
      storage/maria/trnman_public.h:
        new functions
      e27890ca
    • unknown's avatar
      Added maria_commit() and maria_begin() to be used with external tests · f7b766c0
      unknown authored
      Now ma_test1 -M -T and ma_test2 -M -T produces readable, applyable logs
      
      Note: The .MAD file is not binary identical after applying redo compare to a an original file.
      (This is becasue we don't have full information which function called PURGE_REDO_BLOCKS).
      To verify if a file was correctly applied, we now instead compare row checksums
      
      
      BitKeeper/etc/ignore:
        added storage/maria/tmp/*
      include/maria.h:
        Added maria_commit() and maria_begin() to be used with external tests
      storage/maria/ha_maria.cc:
        Ensure maria_def. is read in C mode
      storage/maria/ma_blockrec.c:
        Fixed redo handling.
        _ma_apply_redo_purge_blocks() updated to handle any number of purged blocks
        Removed code to make data file idenitcal after redo (can't easily be done). See changeset comments
        
        Now ma_test1 -M -T and ma_test2 -M -T produces readable, applyable logs
      storage/maria/ma_commit.c:
        More DBUG statements
        Moved variable declaration to start of function (portability fix)
        Added helper functions 'maria_commit()' and 'maria_begin()'
      storage/maria/ma_loghandler.c:
        Fixed wrong REDO_PURGE_BLOCKS initialization
      storage/maria/ma_recovery.c:
        Added UNDO_ROW_UPDATE
        Removed wrong setting of lsn (there was no lsn at the used position)
        Fixed REDO_PURGE_BLOCKS to handle any number of blocks
      storage/maria/ma_test1.c:
        Added transaction support (via maria_begin() & maria_commit()) to get a log that can be applied with maria_read_log
      storage/maria/ma_test2.c:
        Added transaction support (via maria_begin() & maria_commit()) to get a log that can be applied with maria_read_log
      storage/maria/ma_test_recovery:
        Create temporary files in maria/tmp
        Verify files with checksums instead of byte comparisons
      storage/maria/maria_chk.c:
        When using with -dss we only get  filename, records and checksum.
        This is useful to do a quick comparision if a files is identical to another one.
      storage/maria/maria_def.h:
        Added ma_commit()
      storage/maria/maria_read_log.c:
        Added --help
      f7b766c0