An error occurred fetching the project authors.
  1. 12 Dec, 2007 2 commits
    • unknown's avatar
      Removed MARIA_BASE min_row_length (duplicate of min_block_length) · 8224c76f
      unknown authored
      Cleanup of recent code changes in dbug and my_thr_init
      Added name for each safe_mutex (for better DBUG and error reporting)
      Fixed that sort_info.max_records is calculated correctly. This fixed a bug in maria_chk
      Removed duplicate printing of mutex address in dbug log
      
      
      dbug/dbug.c:
        Cleanup of recent code changes
      include/my_pthread.h:
        Added name for each safe_mutex (for better DBUG and error reporting)
      mysys/my_thr_init.c:
        Cleanup of recent code changes
      mysys/thr_mutex.c:
        Added name for each safe_mutex (for better DBUG and error reporting)
      mysys/wqueue.c:
        Removed some mutex printing (as it's done now when we take mutex)
      storage/maria/Makefile.am:
        Fixed that 'make tags' works with xemacs
      storage/maria/ma_blockrec.c:
        base.min_row_length -> base.min_block_length
        (As they where basicly the same variable)
      storage/maria/ma_check.c:
        Moved more common stuff to initialize_variables_for_repair
        Fixed that sort_info.max_records is calculated correctly. This fixed a bug in maria_chk
      storage/maria/ma_create.c:
        More comments
        Fixed that min_pack_length is calculated more correctly
        Removed duplicate variable base.min_row_length
      storage/maria/ma_loghandler.c:
        Removed duplicate printing of mutex address
      storage/maria/ma_open.c:
        Removed base.min_row_length
      storage/maria/ma_packrec.c:
        Removed not anymore needed code
        (One should not change any .base variables as this will affect repair with unpack)
      storage/maria/maria_def.h:
        Removed base.min_row_length
      8224c76f
    • unknown's avatar
      WL#3072 - Maria Recovery: recovery of state.auto_increment. · f7df9add
      unknown authored
      When we log UNDO_KEY_INSERT for an auto_inc key, we update
      state.auto_increment (not anymore at the end of maria_write() except
      if this is a non-transactional table). When Recovery sees UNDO_KEY_INSERT
      in the REDO phase, it reads the auto_inc value from it and updates
      state.auto_increment.
      
      
      mysql-test/r/maria-recovery.result:
        Without the code fix, there would be in CHECK TABLE:
        "Auto-increment value: 0 is smaller than max used value: 3"
        and no AUTO_INCREMENT= clause in SHOW CREATE TABLE.
      mysql-test/t/maria-recovery.test:
        Test of recovery of state.auto_increment: from an old table,
        does the replaying of the log set state.auto_increment to
        what it should be.
      storage/maria/ma_check.c:
        new way of calling ma_retrieve_auto_increment(): pass key
      storage/maria/ma_key.c:
        ma_retrieve_auto_increment() now operates directly with a pointer
        to the key and not on the record.
      storage/maria/ma_key_recover.c:
        dedicated write_hook_for_undo_key_insert(): sets state.auto_increment
        under log's mutex.
      storage/maria/ma_key_recover.h:
        Dedicated hook for UNDO_KEY_INSERT, to set state.auto_increment.
        Such hook needs a new member st_msg_write_hook_for_undo_key::auto_increment,
        which contains the auto_increment value inserted.
      storage/maria/ma_loghandler.c:
        UNDO_KEY_INSERT gets a dedicated write_hook, to set auto_increment.
      storage/maria/ma_recovery.c:
        When in the REDO phase we see UNDO_KEY_INSERT: if the state is older
        than this record, and the key is the auto_increment one, read
        the key's value from the log record and update state.auto_increment.
      storage/maria/ma_test_all.sh:
        use $maria_path to be able to run from /dev/shm (faster)
      storage/maria/ma_update.c:
        bool is more of C++, using my_bool.
        If table is transactional, state.auto_increment is already updated
        in write_hook_for_undo_key_insert().
      storage/maria/ma_write.c:
        If table is transactional, state.auto_increment is not updated at
        the end of maria_write() but rather in write_hook_for_undo_key_insert()
        (under log's mutex, so that a concurrent checkpoint does not read
        state.auto_increment while it is changing - corrupted).
        _ma_ck_write_btree_with_log() extracts the auto_increment value
        from the key, puts it into msg.auto_increment, and this is passed
        to write_hook_for_undo_key_insert().
      storage/maria/maria_def.h:
        change of prototype of ma_retrieve_auto_increment()
      storage/maria/maria_read_log.c:
        use default log file size. Use separate page caches for table
        and logs (needed if maria_block_size!=TRANSLOG_PAGE_SIZE).
      f7df9add
  2. 11 Dec, 2007 1 commit
    • unknown's avatar
      WL#3072 Maria recovery: small fixes to ma_test_recovery · 5051a8e0
      unknown authored
      (output was not repeatable accross machines).
      
      
      storage/maria/ma_check.c:
        comment
      storage/maria/ma_test_recovery.expected:
        output update
      storage/maria/ma_test_recovery:
        * data_file_length/key_file_length differ between machines because
        ma_test2 uses srand(). Removing them from output.
        * there is no reason to copy/restore logs, test should work without
        * applying log to recreate tables should not change log, testing this.
      5051a8e0
  3. 10 Dec, 2007 1 commit
    • unknown's avatar
      Added MARIA_SHARE *share to a lot of places to make code simpler · 2f6f08ed
      unknown authored
      Changed info->s -> share to get more efficent code
      Updated arguments to page accessor functions to use MARIA_SHARE * instead of MARIA_HA *.
      Tested running tests in quick mode (no balance page on insert and only when critical on delete)
      Fixed bug in underflow handling in quick mode
      Fixed bug in log handler where it accessed not initialized variable
      Fixed bug in log handler where it didn't free mutex in unlikely error condition
      Removed double write of page in case of of some underflow conditions
      Added DBUG_PRINT in safemutex lock/unlock
      
      
      dbug/dbug.c:
        Compile without SAFE_MUTEX (to be able to use DBUG_PRINT in safe_mutex code)
        Use calls to get/set my_thread_var->dbug. (Make dbug independent of compile time options for mysys)
      include/my_pthread.h:
        Added prototypes for my_thread_var_get_dbug() & my_thread_var_set_dbug()
      mysql-test/lib/mtr_report.pl:
        Don't check warnings in log files if we are using --extern
      mysys/my_thr_init.c:
        Added my_thread_var_get_dbug() & my_thread_var_set_dbug()
      mysys/thr_mutex.c:
        Added DBUG printing of addresses to mutex for lock/unlock
      storage/maria/ma_blockrec.c:
        Fixed comment
      storage/maria/ma_check.c:
        Added MARIA_SHARE *share to a lot of places to make code simpler
        info->s -> share
        Updated arguments to page accessor functions
      storage/maria/ma_close.c:
        Indentation fixes
      storage/maria/ma_create.c:
        Calculate min_key_length correctly
      storage/maria/ma_dbug.c:
        Indentation fixes
      storage/maria/ma_delete.c:
        Added MARIA_SHARE *share to a lot of places to make code simpler
        info->s -> share
        Updated arguments to page accessor functions
        Removed some writing of key pages that underflow (will be written by caller)
        Fixed crashing bug in underflow handling when using quick mode
      storage/maria/ma_delete_all.c:
        Indentation fixes
      storage/maria/ma_dynrec.c:
        Indentation fixes
      storage/maria/ma_extra.c:
        Fixed indentation
        Removed old useless code
        Reset share->changed if we have written state
      storage/maria/ma_ft_update.c:
        Added MARIA_SHARE *share to a lot of places to make code simpler
        info->s -> share
        Updated arguments to page accessor functions
      storage/maria/ma_info.c:
        Indentation fixes
      storage/maria/ma_key_recover.c:
        Added MARIA_SHARE *share to a lot of places to make code simpler
        info->s -> share
        Updated arguments to page accessor functions
      storage/maria/ma_locking.c:
        Indentation fixes
      storage/maria/ma_loghandler.c:
        Removed wrapper functions translog_mutex_lock and translog_mutex_unlock (safemutex now does same kind of printing)
        Renamed LOGREC_REDO_INSERT_ROW_BLOB to LOGREC_REDO_INSERT_NOT_USED to mark it free
        Fixed some DBUG_PRINT to ensure that convert-dbug-for-diff works
        Fixed bug in translog_flush() that caused log to stop syncing to disk
        Added missing mutex_unlock in case of error
      storage/maria/ma_loghandler.h:
        Renamed LOGREC_REDO_INSERT_ROW_BLOB to LOGREC_REDO_INSERT_NOT_USED to mark it free
      storage/maria/ma_open.c:
        Indentation fixes
      storage/maria/ma_packrec.c:
        Indentation fixes
      storage/maria/ma_page.c:
        Added MARIA_SHARE *share to a lot of places to make code simpler
        info->s -> share
        Updated arguments to page accessor functions
        Added check that we never write a key page without content (except in recovery where a key page may temporary be without content)
      storage/maria/ma_preload.c:
        Updated arguments to page accessor functions
      storage/maria/ma_range.c:
        Updated arguments to page accessor functions
      storage/maria/ma_rkey.c:
        Indentation fixes
      storage/maria/ma_rprev.c:
        Indentation fixes
      storage/maria/ma_rt_index.c:
        Added MARIA_SHARE *share to a lot of places to make code simpler
        info->s -> share
        Updated arguments to page accessor functions
      storage/maria/ma_rt_index.h:
        Updated arguments to page accessor functions
      storage/maria/ma_rt_key.c:
        Added MARIA_SHARE *share to a lot of places to make code simpler
        info->s -> share
        Updated arguments to page accessor functions
      storage/maria/ma_rt_mbr.c:
        Added MARIA_SHARE *share to a lot of places to make code simpler
        info->s -> share
        Updated arguments to page accessor functions
      storage/maria/ma_rt_split.c:
        Added MARIA_SHARE *share to a lot of places to make code simpler
        info->s -> share
        Updated arguments to page accessor functions
      storage/maria/ma_search.c:
        Added MARIA_SHARE *share to a lot of places to make code simpler
        info->s -> share
        Updated arguments to page accessor functions
      storage/maria/ma_sort.c:
        Indentation fixes
      storage/maria/ma_statrec.c:
        Indentation fixes
      storage/maria/ma_test1.c:
        Added extra undo test
        Flush also keys in -u1, to ensure that the full log is flushed
      storage/maria/ma_test2.c:
        Added extra undo test
        Flush also keys in -u1, to ensure that the full log is flushed
      storage/maria/ma_test_recovery.expected:
        Updated results
      storage/maria/ma_test_recovery:
        Added extra undo test
      storage/maria/ma_update.c:
        Indentation fixes
      storage/maria/ma_write.c:
        Added MARIA_SHARE *share to a lot of places to make code simpler
        info->s -> share
        Updated arguments to page accessor functions
        Prepare for quick mode for insert (don't balance page)
      storage/maria/maria_chk.c:
        Added MARIA_SHARE *share to a lot of places to make code simpler
        info->s -> share
        Updated arguments to page accessor functions
      storage/maria/maria_def.h:
        Updated arguments to page accessor functions
      2f6f08ed
  4. 04 Dec, 2007 1 commit
    • unknown's avatar
      Added error HA_ERR_FILE_TOO_SHORT to be used when files are shorter than... · ebf7ab7b
      unknown authored
      Added error HA_ERR_FILE_TOO_SHORT to be used when files are shorter than expected (by my_read/my_pread)
      Added debugger hook _my_dbug_put_break_here() that is called if we get a CRC that matches --debug-crc-break (my_crc_dbug_break)
      Fixed REDO_REPAIR to use all repair modes (repair, repair_by_sort, repair_paralell
      REDO_REPAIR now also logs used key map
      Fixed some bugs in REDO logging of key pages
      Better error messages from maria_read_log
      Added my_readwrite_flags to init_pagecache() to be able to get better error messages and simplify code.
      Don't allow pagecaches with less than 8 blocks (Causes strange crashes)
      Added EXTRA_DEBUG_KEY_CHANGES. When this is defined some REDO_INDEX entries contains page checksums (these are calculated and checked in DBUG mode, ignored otherwise)
      Fixed bug in ma_pagecache unit tests that caused program to sometimes fail
      Added some missing calls to MY_INIT() that caused some unit tests to fail
      Fixed that TRUNCATE works properly on temporary MyISAM files
      Updates some result files to new table checksums results (checksum when NULL fields are ignored)
      perl test-insert can be replayed with maria_read_log!
      
      
      sql/share/Makefile.am:
        Change mode to -rw-rw-r--
      BitKeeper/etc/ignore:
        added storage/maria/unittest/page_cache_test_file_1 storage/maria/unittest/pagecache_debug.log
      include/maria.h:
        Added maria_tmpdir
      include/my_base.h:
        Added error HA_ERR_FILE_TOO_SHORT
      include/my_sys.h:
        Added variable my_crc_dbug_check
        Added function my_dbug_put_break_here()
      include/myisamchk.h:
        Added org_key_map (Needed for writing REDO record for REPAIR)
      mysql-test/r/innodb.result:
        Updated to new checksum algorithm (NULL ignored)
      mysql-test/r/mix2_myisam.result:
        Updated to new checksum algorithm (NULL ignored)
      mysql-test/r/myisam.result:
        Updated to new checksum algorithm (NULL ignored)
      mysql-test/t/myisam.test:
        Added used table
      mysys/checksum.c:
        Added DBUG for checksum results
        Added debugger hook so that _my_dbug_put_break_here() is called if we get matching CRC
      mysys/lf_alloc-pin.c:
        Fixed compiler warning
      mysys/my_handler.c:
        Added new error message
      mysys/my_init.c:
        If my_progname is not given, use 'unknown' form my_progname_short
        Added debugger function my_debug_put_break_here()
      mysys/my_pread.c:
        In case of too short file when MY_NABP or MY_FNABP is specified, give error HA_ERR_FILE_TO_SHORT
      mysys/my_read.c:
        In case of too short file when MY_NABP or MY_FNABP is specified, give error HA_ERR_FILE_TO_SHORT
      sql/mysqld.cc:
        Added debug option --debug-crc-break
      sql/sql_parse.cc:
        Trivial optimization
      storage/maria/ha_maria.cc:
        Renamed variable to be more logical
        Ensure that param.testflag is correct when calling repair
        Added extra argument to init_pagecache
        Set default value for maria_tempdir
      storage/maria/ma_blockrec.c:
        Test for HA_ERR_FILE_TOO_SHORT instead for -1
      storage/maria/ma_cache.c:
        Test for HA_ERR_FILE_TOO_SHORT instead for -1
      storage/maria/ma_check.c:
        Set param->testflag to match how repair is run (needed for REDO logging)
        Simple optimization
        Moved flag if page is node from pagelength to keypage-flag byte
        Log used key map in REDO log.
      storage/maria/ma_delete.c:
        Remember previous UNDO entry when writing undo (for future CLR records)
        Moved flag if page is node from pagelength to keypage-flag byte
        Fixed some bugs in redo logging
        Added CRC for some translog REDO_INDEX entries
      storage/maria/ma_dynrec.c:
        Test for HA_ERR_FILE_TOO_SHORT instead for -1
      storage/maria/ma_ft_update.c:
        Fixed call to _ma_store_page_used()
      storage/maria/ma_key_recover.c:
        Added CRC for some translog REDO_INDEX entries
        Removed not needed pagecache_write() in _ma_apply_redo_index()
      storage/maria/ma_locking.c:
        Test for HA_ERR_FILE_TOO_SHORT instead for -1
      storage/maria/ma_loghandler.c:
        Added used key map to REDO_REPAIR_TABLE
      storage/maria/ma_loghandler.h:
        Added operation for checksum of key pages
      storage/maria/ma_open.c:
        Allocate storage for undo lsn pointers
      storage/maria/ma_pagecache.c:
        Remove not needed include file
        Change logging to use fd: for file descritors as other code
        Added my_readwrite_flags to init_pagecache() to be able to get better error messages for maria_chk/maria_read_log
        Don't allow pagecaches with less than 8 blocks
        Remove wrong DBUG_ASSERT()
      storage/maria/ma_pagecache.h:
        Added readwrite_flags
      storage/maria/ma_recovery.c:
        Better error messages for maria_read_log:
        - Added eprint() for printing error messages
        - Print extra \n before error message if we are printing %0 %10 ...
        
        Added used key_map to REDO_REPAIR log entry
        More DBUG
        Call same repair method that was used by mysqld
      storage/maria/ma_rt_index.c:
        Moved flag if page is node from pagelength to keypage-flag byte
      storage/maria/ma_rt_key.c:
        Fixed call to _ma_store_page_used()
      storage/maria/ma_rt_split.c:
        Moved flag if page is node from pagelength to keypage-flag byte
      storage/maria/ma_static.c:
        Added maria_tmpdir
      storage/maria/ma_test1.c:
        Updated call to init_pagecache()
      storage/maria/ma_test2.c:
        Updated call to init_pagecache()
      storage/maria/ma_test3.c:
        Updated call to init_pagecache()
      storage/maria/ma_write.c:
        Removed #ifdef NOT_YET
        Moved flag if page is node from pagelength to keypage-flag byte
        Fixed bug in  _ma_log_del_prefix()
      storage/maria/maria_chk.c:
        Fixed wrong min limit for page_buffer_size
        Updated call to init_pagecache()
      storage/maria/maria_def.h:
        Added EXTRA_DEBUG_KEY_CHANGES. When this is defined some REDO_INDEX entries contains page checksums
        Moved flag if page is node from pagelength to keypage-flag byte
      storage/maria/maria_ftdump.c:
        Updated call to init_pagecache()
      storage/maria/maria_pack.c:
        Updated call to init_pagecache()
        Reset share->state.create_rename_lsn & share->state.is_of_horizon
      storage/maria/maria_read_log.c:
        Better error messages
        Added --tmpdir option (needed to set temporary directory for REDO_REPAIR)
        Added --start-from-lsn
        Changed option for --display-only to 'd' (wanted to use -o for 'offset')
      storage/maria/unittest/lockman2-t.c:
        Added missing call to MY_INIT()
      storage/maria/unittest/ma_pagecache_consist.c:
        Updated call to init_pagecache()
      storage/maria/unittest/ma_pagecache_single.c:
        Fixed bug that caused program to sometimes fail
        Added some DBUG_ASSERTS()
        Changed some calls to malloc()/free() to my_malloc()/my_free()
        Create extra file to expose original hard-to-find bug
      storage/maria/unittest/ma_test_loghandler-t.c:
        Updated call to init_pagecache()
      storage/maria/unittest/ma_test_loghandler_first_lsn-t.c:
        Updated call to init_pagecache()
      storage/maria/unittest/ma_test_loghandler_max_lsn-t.c:
        Updated call to init_pagecache()
      storage/maria/unittest/ma_test_loghandler_multigroup-t.c:
        Updated call to init_pagecache()
      storage/maria/unittest/ma_test_loghandler_multithread-t.c:
        Updated call to init_pagecache()
      storage/maria/unittest/ma_test_loghandler_noflush-t.c:
        Updated call to init_pagecache()
      storage/maria/unittest/ma_test_loghandler_pagecache-t.c:
        Updated call to init_pagecache()
      storage/maria/unittest/ma_test_loghandler_purge-t.c:
        Updated call to init_pagecache()
      storage/maria/unittest/test_file.c:
        Changed malloc()/free() to my_malloc()/my_free()
        Fixed memory leak
        Changd logic a bit while trying to find bug in reset_file()
      storage/maria/unittest/trnman-t.c:
        Added missing call to MY_INIT()
      storage/myisam/mi_cache.c:
        Test for HA_ERR_FILE_TOO_SHORT instead for -1
      storage/myisam/mi_create.c:
        Removed O_EXCL to get TRUNCATE to work for temporary files
      storage/myisam/mi_dynrec.c:
        Test for HA_ERR_FILE_TOO_SHORT instead for -1
      storage/myisam/mi_locking.c:
        Test for HA_ERR_FILE_TOO_SHORT instead for -1
      mysql-test/r/old-mode.result:
        New BitKeeper file ``mysql-test/r/old-mode.result''
      mysql-test/t/old-mode-master.opt:
        New BitKeeper file ``mysql-test/t/old-mode-master.opt''
      mysql-test/t/old-mode.test:
        New BitKeeper file ``mysql-test/t/old-mode.test''
      ebf7ab7b
  5. 28 Nov, 2007 1 commit
    • unknown's avatar
      Fixed repair_by_sort to work with BLOCK_RECORD · 4e0964cb
      unknown authored
      Fixed bugs in undo logging
      Fixed bug where head block was split before min_row_length (caused Maria to believe row was crashed on read)
      Reserved place for reference-transid on key pages (for packing of transids)
      ALTER TABLE and INSERT ... SELECT now uses fast creation of index
          
      Known bugs:
      ma_test_recovery fails because of a bug in redo handling when log is cut directly after a redo (Guilhem knows how to fix)
      ma_test_recovery.excepted is not totally correct, because of the above bug
      mysqld sometimes fails to restart; Fails with error "end_of_redo_phase: Assertion `long_trid != 0' failed"; Guilhem to investigate
      
      
      include/maria.h:
        Prototype changes
        Added current_filepos to st_maria_sort_info
      mysql-test/r/maria.result:
        Updated results that changes as alter table and insert ... select now uses fast creation of index
      mysys/mf_iocache.c:
        Reset variable to gurard against double invocation
      storage/maria/ma_bitmap.c:
        Added _ma_bitmap_reset_cache() (needed for repair)
      storage/maria/ma_blockrec.c:
        Simplify code
        More initial allocations
        Fixed bug where head block was split before min_row_length (caused Maria to believe row was crashed on read)
      storage/maria/ma_blockrec.h:
        Moved TRANSID_SIZE to maria_def.h
        Added prototype for new functions
      storage/maria/ma_check.c:
        Simplicy code
        Fixed repair_by_sort to work with BLOCK_RECORD
        - When using BLOCK_RECORD or UNPACK create new Maria handle
        - Use common initializer function
        - Align code with maria_repair()
        
        Made some changes to maria_repair_parallel() to use common initializer function
        Removed ASK_MONTY section by fixing noted problem
      storage/maria/ma_close.c:
        Moved check for readonly to _ma_state_info_write()
      storage/maria/ma_key_recover.c:
        Use different log entries if key root changes or not.
        This fixed some bugs when tree grows
      storage/maria/ma_key_recover.h:
        Added keynr to st_msg_to_write_hook_for_undo_key
      storage/maria/ma_loghandler.c:
        Added INIT_LOGREC_UNDO_KEY_INSERT_WITH_ROOT
      storage/maria/ma_loghandler.h:
        Added INIT_LOGREC_UNDO_KEY_INSERT_WITH_ROOT
      storage/maria/ma_open.c:
        Added TRANSID to all key pages (for future compressing of trans id's)
        For compressed records, alloc a bit bigger buffer to avoid valgrind warnings
        If table is opened readonly, don't update state
      storage/maria/ma_packrec.c:
        Allocate bigger array for bit unpacking to avoid valgrind errors
      storage/maria/ma_recovery.c:
        Added UNDO_KEY_INSERT_WITH_ROOT & UNDO_KEY_DELETE_WITH_ROOT
      storage/maria/ma_sort.c:
        More logging
      storage/maria/ma_test_all.sh:
        More tests
      storage/maria/ma_test_recovery.expected:
        Update results
        Note that this is not complete becasue of a bug in recovery
      storage/maria/ma_test_recovery:
        Removed recreation of index (not needed when we have redo for index pages)
      storage/maria/maria_chk.c:
        When using flag --read-only, don't update status for files
        When using --unpack, don't use REPAIR_BY_SORT if other repair option is given
        Enable repair_by_sort for BLOCK records
        Removed not needed newline at start of --describe
      storage/maria/maria_def.h:
        Support for TRANSID_SIZE to key pages
      storage/maria/maria_read_log.c:
        renamed --only-display to --display-only
      4e0964cb
  6. 20 Nov, 2007 1 commit
    • unknown's avatar
      Fixes for redo/undo logging of key pages · 6b3743f0
      unknown authored
      New extendable format for maria_log_control file
      Fixed some compiler warnings
      
      
      include/maria.h:
        Added maria_disable_logging() and maria_enable_logging()
      mysql-test/include/maria_verify_recovery.inc:
        Updated tests now when key redo/undo works
      mysql-test/r/maria-recovery.result:
        Updated tests now when key redo/undo works
      storage/maria/ma_blockrec.c:
        Use unified CLR code
        Added rec_lsn for full pages
        Moved clr write hook to ma_key_recover.c
        Changed REDO code to keep pages pinned until undo
        Mark page_link's as changed
      storage/maria/ma_blockrec.h:
        Moved write_hook_for_clr_end() to ma_key_recover.c
      storage/maria/ma_check.c:
        Changed key check code to use PAGECACHE_READ_UNKNOWN_PAGE
        Fixed wrong warning when checking files after maria_pack
        When unpacking files, we have to use new keypos_to_recpos method
        When doing repair, we can disregard index key file pages in page cache
      storage/maria/ma_commit.c:
        Added simple enable/disable logging functions
        (Needed for recovery)
      storage/maria/ma_control_file.c:
        Make maria control file extendable without having to make it incompatible for older versions
      storage/maria/ma_control_file.h:
        New error messages
        Added CONTROL_FILE_VERSION
      storage/maria/ma_delete.c:
        Added redo/undo for key pages
        change_length -> changed_length to make things similar
        More comments & more DBUG
      storage/maria/ma_key_recover.c:
        Unified CLR method
        Moved here write_hook_for_clr_end() and common keypage log functions
        Changed REDO to keep pages pinned until undo
        Changed UNDO code to change key_root under log mutex
      storage/maria/ma_key_recover.h:
        New structures and functions
      storage/maria/ma_loghandler.c:
        Include needed files
      storage/maria/ma_open.c:
        Change maria_open() to use pread() instead of read()
      storage/maria/ma_page.c:
        Fixed bug in key_del handling
        Clear pages if IDENTICAL_PAGES_AFTER_RECOVERY is defined
      storage/maria/ma_pagecache.c:
        Indentation and spelling fixes
        More DBUG
        Added helper function: pagecache_block_link_to_buffer()
      storage/maria/ma_pagecache.h:
        Added pagecache_block_link_to_buffer()
      storage/maria/ma_recovery.c:
        Fixed state.changed
        Fixed that REDO keeps pages pinned until UNDO
        Some bug fixes from previous commit
        Fixes for UNDO/REDO of key pages
      storage/maria/ma_search.c:
        Fixed packing and storing of keys to provide more information to caller so
        that we can do efficent REDO logging of the changes.
      storage/maria/ma_test1.c:
        Fixed bug with not initialized variable
      storage/maria/ma_test2.c:
        Removed not used code
      storage/maria/ma_test_all.res:
        Updated results
      storage/maria/ma_test_all.sh:
        Changed one test to test more
        Removed timing tests as not relevant here
      storage/maria/ma_test_recovery.expected:
        Updated test result after redo/undo if key pages works
      storage/maria/ma_test_recovery:
        Updated test after redo/undo if key pages works
      storage/maria/ma_write.c:
        Moved some general log functions to ma_key_recover.c
        Fixed some bugs in undo
        Moved ma_log_split() to _ma_split_page()
        Small changes in some function arguments to be able to do redo logging
      storage/maria/maria_chk.c:
        disable logging while doing repair table
      storage/maria/maria_def.h:
        New function prototypes
        Move some structs and functions to ma_key_recover.c
      storage/maria/unittest/ma_control_file-t.c:
        Updated with patch from Sanja
        NOTE: This is not complete and need to be updated to new control file format
      storage/maria/unittest/ma_test_loghandler-t.c:
        Fixed compiler warning
      6b3743f0
  7. 14 Nov, 2007 1 commit
    • unknown's avatar
      First part of redo/undo for key pages · 21fd2a5a
      unknown authored
      Added key_nr to st_maria_keydef for faster keyinfo->keynr conversion
      For transactional tables, shift record number in keys up with 1 bit to have place to indicate if transid follows
      Checksum for MyISAM now ignores NULL and not used part of VARCHAR
      Renamed some variables that caused shadow compiler warnings
      Moved extra() call when waiting for tables to not be used to after tables are removed from cache.
      Fixed crashing bugs when using Maria TEMPORARY tables with TRUNCATE. Removed 'hack' code in sql directory to go around this bug.
      pagecache_unlock_by_ulink() now has extra argument to say if page was changed.
      Give error message if we fail to open control file
      Mark page cache variables as not flushable
      
      
      include/maria.h:
        Made min page cache larger (needed for pinning key page)
        Added key_nr to st_maria_keydef for faster keyinfo->keynr conversion
        Added write_comp_flag to move some runtime code to maria_open()
      include/my_base.h:
        Added new error message to be used when handler initialization failed
      include/my_global.h:
        Renamed dummy to swap_dummy to avoid conflicts with local 'dummy' variables
      include/my_handler.h:
        Added const to some parameters
      mysys/array.c:
        More DBUG
      mysys/my_error.c:
        Fixed indentation
      mysys/my_handler.c:
        Added const to some parameters
        Added missing error messages
      sql/field.h:
        Renamed variables to avoid variable shadowing
      sql/handler.h:
        Renamed parameter to avoid variable name conflict
      sql/item.h:
        Renamed variables to avoid variable shadowing
      sql/log_event_old.h:
        Renamed variables to avoid variable shadowing
      sql/set_var.h:
        Renamed variables to avoid variable shadowing
      sql/sql_delete.cc:
        Removed maria hack for temporary tables
        Fixed indentation
      sql/sql_table.cc:
        Moved extra() call when waiting for tables to not be used to after tables are removed from cache.
        This was needed to ensure we don't do a PREPARE_FOR_DROP or similar call while the table is still in use.
      sql/table.cc:
        Copy page_checksum from share
        Removed Maria hack
      storage/maria/Makefile.am:
        Added new files
      storage/maria/ha_maria.cc:
        Renamed records -> record_count and info -> create_info to avoid variable name conflicts
        Mark page cache variables as not flushable
      storage/maria/ma_blockrec.c:
        Moved _ma_unpin_all_pages() to ma_key_recover.c
        Moved init of info->pinned_pages to ma_open.c
        Moved _ma_finalize_row() to maria_key_recover.h
        Renamed some variables to avoid variable name conflicts
        Mark page_link.changed for blocks we change directly
        Simplify handling of undo link when writing LOGREC_UNDO_ROW_INSERT (old code crashed when having redo for index)
      storage/maria/ma_blockrec.h:
        Removed extra empty line
      storage/maria/ma_checkpoint.c:
        Remove not needed trnman.h
      storage/maria/ma_close.c:
        Free pinned pages (which are now always allocated)
      storage/maria/ma_control_file.c:
        Give error message if we fail to open control file
      storage/maria/ma_delete.c:
        Changes for redo logging (first part, logging of underflow not yet done)
        - Log undo-key-delete
        - Log delete of key
        - Updated arguments to _ma_fetch_keypage(), _ma_dispose(), _ma_write_keypage(), _ma_insert()
        - Added new arguments to some functions to be able to write redo information
        - Mark key pages as changed when we write with PAGECACHE_LOCK_LEFT_WRITELOCKED
        
        Remove one not needed _ma_write_keypage() in d_search() when upper level will do the write anyway
        Changed 2 bmove_upp() to bmove() as this made code easer to understand
        More function comments
        Indentation fixes
      storage/maria/ma_ft_update.c:
        New arguments to _ma_write_keypage()
      storage/maria/ma_loghandler.c:
        Fixed some DBUG_PRINT messages
        Simplify code
        Added new log entrys for key page redo
        Renamed some variables to avoid variable name shadowing
      storage/maria/ma_loghandler.h:
        Moved some defines here
        Added define for storing key number on key pages
        Added new translog record types
        Added enum for type of operations in LOGREC_REDO_INDEX
      storage/maria/ma_open.c:
        Always allocate info.pinned_pages (we need now also for normal key page usage)
        Update keyinfo->key_nr
        Added virtual functions to convert record position o number to be stored on key pages
        Update keyinfo->write_comp_flag to value of search flag to be used when writing key
      storage/maria/ma_page.c:
        Added redo for key pages
        - Extended _ma_fetch_keypage() with type of lock to put on page and address to used MARIA_PINNED_PAGE
        - _ma_fetch_keypage() now pin's pages if needed
        - Extended _ma_write_keypage() with type of locks to be used
        - ma_dispose() now locks info->s->state.key_del from other threads
        - ma_dispose() writes redo log record
        - ma_new() locks info->s->state.key_del from other threads if it was used
        - ma_new() now pins read page
        
        Other things:
        - Removed some not needed arguments from _ma_new() and _ma_dispose)
        - Added some new variables to simplify code
        - If EXTRA_DEBUG is used, do crc on full page to catch not unitialized bytes
      storage/maria/ma_pagecache.h:
        Applied patch from Sanja to add extra argument to pagecache_unlock_by_ulink() to mark if page was changed
        Added some defines for pagecache priority levels that one can use
      storage/maria/ma_range.c:
        Added new arguments for call to _ma_fetch_keypage()
      storage/maria/ma_recovery.c:
        - Added hooks for new translog types:
          REDO_INDEX, REDO_INDEX_NEW_PAGE, REDO_INDEX_FREE_PAGE, UNDO_KEY_INSERT, UNDO_KEY_DELETE and
          UNDO_KEY_DELETE_WITH_ROOT.
        - Moved variable declarations to start of function (portability fixes)
        - Removed some not needed initializations
        - Set only relevant state changes for each redo/undo entry
      storage/maria/lockman.c:
        Removed end space
      storage/maria/ma_check.c:
        Removed end space
      storage/maria/ma_create.c:
        Removed end space
      storage/maria/ma_locking.c:
        Removed end space
      storage/maria/ma_packrec.c:
        Removed end space
      storage/maria/ma_pagecache.c:
        Removed end space
      storage/maria/ma_panic.c:
        Removed end space
      storage/maria/ma_rt_index.c:
        Added new arguments for call to _ma_fetch_keypage(), _ma_write_keypage(), _ma_dispose() and _ma_new()
        Fixed indentation
      storage/maria/ma_rt_key.c:
        Added new arguments for call to _ma_fetch_keypage()
      storage/maria/ma_rt_split.c:
        Added new arguments for call to _ma_new()
        Use new keypage header
        Added new arguments for call to _ma_write_keypage()
      storage/maria/ma_search.c:
        Updated comments & indentation
        Added new arguments for call to _ma_fetch_keypage()
        Made some variables and arguments const
        Added virtual functions for converting row position to number to be stored in key
        use MARIA_RECORD_POS of record position instead of my_off_t
        Record in MARIA_KEY_PARAM how page was changed one key insert (needed for REDO)
      storage/maria/ma_sort.c:
        Removed end space
      storage/maria/ma_statrec.c:
        Updated arguments for call to _ma_rec_pos()
      storage/maria/ma_test1.c:
        Fixed too small buffer to init_pagecache()
        Fixed bug when using insert_count and test_flag
      storage/maria/ma_test2.c:
        Use more resonable pagecache size
        Remove not used code
        Reset blob_length to fix wrong output message
      storage/maria/ma_test_all.sh:
        Fixed wrong test
      storage/maria/ma_write.c:
        Lots of new code to handle REDO of key pages
        No logic changes because of REDO code, mostly adding new arguments and adding new code for logging 
        
        Added new arguments for calls to _ma_fetch_keypage(), _ma_write_keypage() and similar functions
        Move setting of comp_flag in ma_ck_wrte_btree() from runtime to maria_open()
        Zerofill new used pages for:
        - To remove possible sensitive data left in buffer
        - To get idenitical data on pages after running redo
        - Better compression of pages if archived
      storage/maria/maria_chk.c:
        Added information if table is crash safe
      storage/maria/maria_def.h:
        New virtual function to convert between record position on key and normal record position
        Aded mutex and extra variables to handle locking of share->state.key_del
        Moved some structure variables to get things more aligned
        Added extra arguments to MARIA_KEY_PARAM to be able to remember what was changed on key page on key insert
        Added argument to MARIA_PINNED_PAGE to indicate if page was changed
        Updated prototypes for functions
        Added some structures for signaling changes in REDO handling
      storage/maria/unittest/ma_pagecache_single.c:
        Updated arguments for changed function calls
      storage/myisam/mi_check.c:
        Made calc_check_checksum virtual
      storage/myisam/mi_checksum.c:
        Update checksums to ignore null columns
      storage/myisam/mi_create.c:
        Mark if table has null column (to know when we have to use mi_checksum())
      storage/myisam/mi_open.c:
        Added virtual function for calculating checksum to be able to easily ignore NULL fields
      storage/myisam/mi_test2.c:
        Fixed bug
      storage/myisam/myisamdef.h:
        Added virtual function for calculating checksum during check table
        Removed ha_key_cmp() as this is in handler.h
      storage/maria/ma_key_recover.c:
        New BitKeeper file ``storage/maria/ma_key_recover.c''
      storage/maria/ma_key_recover.h:
        New BitKeeper file ``storage/maria/ma_key_recover.h''
      storage/maria/ma_key_redo.c:
        New BitKeeper file ``storage/maria/ma_key_redo.c''
      21fd2a5a
  8. 19 Oct, 2007 1 commit
    • unknown's avatar
      Merge some changes from sql directory in 5.1 tree · 13d53bf6
      unknown authored
      Changed format for REDO_INSERT_ROWS_BLOBS
      Fixed several bugs in handling of big blobs
      Added redo_free_head_or_tail() & redo_insert_row_blobs()
      Added uuid to control file
      maria_checks now verifies that not used part of bitmap is 0
      REDO_PURGE_BLOCKS -> REDO_FREE_BLOCKS
      Added REDO_FREE_HEAD_OR_TAIL
      Fixes problem when trying to read block outside of file during REDO
      
      
      include/my_global.h:
        STACK_DIRECTION is already set by configure
      mysql-test/r/maria.result:
        Updated results
      mysql-test/t/maria.test:
        Test shrinking of VARCHAR
      mysys/my_realloc.c:
        Fixed indentation
      mysys/safemalloc.c:
        Fixed indentation
      sql/filesort.cc:
        Removed some casts
      sql/mysqld.cc:
        Added missing setting of myisam_stats_method_str
      sql/uniques.cc:
        Removed some casts
      storage/maria/ma_bitmap.c:
        Added printing of bitmap (for debugging)
        Renamed _ma_print_bitmap() -> _ma_print_bitmap_changes()
        Added _ma_set_full_page_bits()
        Fixed bug in ma_bitmap_find_new_place() (affecting updates) when using big files
      storage/maria/ma_blockrec.c:
        Changed format for REDO_INSERT_ROWS_BLOBS
        Fixed several bugs in handling of big blobs
        Added code to fix some cases where redo when using blobs didn't produce idenital .MAD files as normal usage
        REDO_FREE_ROW_BLOCKS doesn't anymore change pages; We only mark things free in bitmap
        Remove TAIL and filler extents from REDO_FREE_BLOCKS log entry. (Fixed some asserts)
        REDO_PURGE_BLOCKS -> REDO_FREE_BLOCKS
        Delete tails in update. (Fixed bug when doing update that shrinks blob/varchar length)
        Fixed bug when doing insert in block outside of file size.
        Added redo_free_head_or_tail() & redo_insert_row_blobs()
        Added pagecache_unlock_by_link() when read fails.
        Much more comments, DBUG and ASSERT entries
      storage/maria/ma_blockrec.h:
        Prototypes of new functions
        Define of SUB_RANGE_SIZE & BLOCK_FILLER_SIZE
      storage/maria/ma_check.c:
        Verify that not used part of bitmap is 0
      storage/maria/ma_control_file.c:
        Added uuid to control file
      storage/maria/ma_loghandler.c:
        REDO_PURGE_BLOCKS -> REDO_FREE_BLOCKS
        Added REDO_FREE_HEAD_OR_TAIL
      storage/maria/ma_loghandler.h:
        REDO_PURGE_BLOCKS -> REDO_FREE_BLOCKS
        Added REDO_FREE_HEAD_OR_TAIL
      storage/maria/ma_pagecache.c:
        If we write full block, remove error flag for block.
        (Fixes problem when trying to read block outside of file)
      storage/maria/ma_recovery.c:
        REDO_PURGE_BLOCKS -> REDO_FREE_BLOCKS
        Added REDO_FREE_HEAD_OR_TAIL
      storage/maria/ma_test1.c:
        Allow option after 'b' to be compatible with ma_test2
        (This is just to simplify test scripts like ma_test_recovery)
      storage/maria/ma_test2.c:
        Default size of blob is now 1000 instead of 1
      storage/maria/ma_test_all.sh:
        Added test for bigger blobs
      storage/maria/ma_test_recovery.expected:
        Updated results
      storage/maria/ma_test_recovery:
        Added test for bigger blobs
      13d53bf6
  9. 09 Oct, 2007 1 commit
    • unknown's avatar
      Moved randomize and my_rnd under mysys · 496741d5
      unknown authored
      Added my_uuid
      Added pre-support for PAGE_CHECKSUM
      Added syntax for CREATE ... PAGE_CHECKSUM=# TABLE_CHECKSUM=#
      Reserved place for page checksums on index, bitmap and block pages
      Added index number to header of index pages
      Added linked list for free directory entries (speeds up inserts with BLOCK format)
      Calculate checksums in original column order (fixes bug with checksum on rows with BLOCK format)
      Cleaned up all index handling to use 'info->s->keypage_header' (variable size) as the header for index pages (before this was '2')
      Added 0xffffffff to end of index and block data bases and 0xfffffffe at end of bitmap pages when page checksums are not enabled
      Added _ma_get_page_used() and _ma_get_used_and_node() to simplify index page header handling
      rec_per_key_part is now in double precision
      Reserved place in index file for my_guid and nulls_per_key_part
      Give error HA_ERR_NEW_FILE if trying to open a Maria file with new, not yet supported extensions
      
      Lots of renames to increase readability:
      
      randomize() -> my_rnd_init()
      st_maria_info -> st_maria_handler
      st_maria_info -> MARIA_HA
      st_maria_isaminfo -> st_maria_info
      rand_struct -> my_rand_struct
      rec_per_key_rows -> records_at_analyze
      
      
      client/mysqladmin.cc:
        rand_struct -> my_rrnd_struct
      include/maria.h:
        st_maria_info -> MARIA_HA
        st_maria_isaminfo -> st_maria_info
        Changed analyze statistics to be of double precission
        Changed offset to field to be 32bits instead of 64 (safe as a record without blobs can't be that big)
      include/my_base.h:
        Added HA_OPTION_PAGE_CHECKSUM & HA_CREATE_PAGE_CHECKSUM
        Fixed comments
        Added HA_ERR_NEW_FILE
      include/my_sys.h:
        Added prototypes and structures for my_uuid() and my_rnd()
      include/myisamchk.h:
        Changed some buffers to size_t
        Added possibility to have key statistics with double precission
      include/mysql_com.h:
        Move rand functions to mysys
      libmysql/Makefile.shared:
        Added my_rnd
      mysql-test/r/maria.result:
        Updated results
      mysql-test/t/maria.test:
        More tests for checksum
      mysys/Makefile.am:
        Added my_rnd.c and my_uuid.c
      server-tools/instance-manager/listener.cc:
        Fixed include order (my_global.h should always be first)
      server-tools/instance-manager/mysql_connection.cc:
        Fixed include order (my_global.h should always be first)
        Use my_rnd_init()
      server-tools/instance-manager/mysql_connection.h:
        rand_struct -> my_rand_struct
      sql/handler.h:
        Added flag for page checksums
      sql/item_func.cc:
        Use new my_rnd() interface
      sql/item_func.h:
        Use new my_rnd() interface
      sql/item_strfunc.cc:
        Use new my_rnd() interface
      sql/lex.h:
        Added PAGE_CHECKSUM and TABLE_CHECKSUM
      sql/mysql_priv.h:
        Use new my_rnd() interface
      sql/mysqld.cc:
        Use new my_rnd() interface
      sql/password.c:
        Move my_rnd() to mysys
        Use new my_rnd() interface
      sql/sql_class.cc:
        Use new my_rnd() interface
      sql/sql_class.h:
        Use new my_rnd() interface
      sql/sql_crypt.cc:
        Use new my_rnd() interface
      sql/sql_crypt.h:
        Use new my_rnd() interface
      sql/sql_show.cc:
        Simpler handling of ha_choice_values
        Added PAGE_CHECKSUM
      sql/sql_table.cc:
        Enable correct checksum handling (for now) if not running in compatible mode
      sql/sql_yacc.yy:
        Added table option PAGE_CHECKSUM
        Added future compatible table option TABLE_CHECKSUM (alias for CHECKSUM)
        Added 'choice' target to simplify code
      sql/table.cc:
        Store flag for PAGE_CHECKSUM
      sql/table.h:
        Added support for PAGE_CHECKSUM
      storage/maria/ha_maria.cc:
        Remove protection for incompatbile frm and MAI
        (Slow, not needed test)
        Rec_per_key is now in double
        Remember row type for table
        Give warning if one Maria uses another row type than requested
        Removed some old ASK_MONTY entries (added comments instead)
        Added handling of PAGE_CHECKSUM flags
      storage/maria/ma_bitmap.c:
        Added page checksums to bitmap pages
        Added special bitmap marker for bitmap pages
        (Used to find bugs when running without page checksums)
      storage/maria/ma_blockrec.c:
        Added a free-link list over directory entries. This makes insert of small rows faster as we don't
        have to scan the whole directory to find a not used entry.
        Moved SANITY_CHECKS to maria_def.h
        Simplify code by introducing dir_entry_pos()
        Added support for PAGE_CHECKSUM
      storage/maria/ma_blockrec.h:
        Added DIR_FREE_SIZE (linked list of free directory entries)
        Added PAGE_CHECKSUM
        Added 'dir_entry_pos()'
      storage/maria/ma_check.c:
        Check that index pages has correct index number
        Calculate rec_per_key with double precission
        Simplify code by using '_ma_get_used_and_node()'
        Check free directory list
        Remove wrong end \n from messages
        maria_data_on_page() -> _ma_get_page_used()
        maria_putint() -> _ma_store_page_used()
        rec_per_key_rows -> records_at_analyze
      storage/maria/ma_checksum.c:
        Calculate checksum in original column order
      storage/maria/ma_create.c:
        Store original column order in index file
        Reserve place for nulls_per_key_part (future)
        Added support for PAGE_CHECKSUM
      storage/maria/ma_dbug.c:
        Fixed wrong debug output of key of type 'ulong'
      storage/maria/ma_delete.c:
        maria_data_on_page() -> _ma_get_used_and_node()
        maria_data_on_page() -> _ma_get_page_used()
        maria_putint() -> _ma_store_page_used()
        Added page header (index key number) to all index pages
        Reserved page for checksum on index pages
        Use keypage_header
      storage/maria/ma_ft_update.c:
        maria_putint() -> _ma_store_page_used()
        Store key number at start of page
      storage/maria/ma_loghandler.h:
        st_maria_info -> MARIA_HA
      storage/maria/ma_open.c:
        rec_per_key is now in double precission
        Added 'nulls_per_key_part'
        Added 'extra_options' (flags for future)
        Added support for PAGE_CHECKSUM
        Give error HA_ERR_NEW_FILE when using unsupported maria extensions
        Added comments
        Add maria_uuid to index file
        Added functions to store and read column_nr map.
        Changed some functions to return my_bool instead of uint
      storage/maria/ma_page.c:
        Added checks that pages has correct key nr
        Store 0xffffffff in checksum position if page checksums are not enabled
        Moved key-page-delete link to take into account keypage header
      storage/maria/ma_preload.c:
        Remove old MyISAM dependent code
        When scanning pages, only add pages to page cache for the requested index
      storage/maria/ma_range.c:
        maria_data_on_page() -> _ma_get_used_and_node()
        Use keypage_header
      storage/maria/ma_rt_index.c:
        Fixed indentation
      storage/maria/ma_rt_index.h:
        Added support for dynamic index page header
        Reserved place for PAGE_CHECKSUM
      storage/maria/ma_rt_key.c:
        Fixed indentation
        maria_data_on_page() -> _ma_get_page_used()
        maria_putint() -> maria_store_page_used()
      storage/maria/ma_rt_mbr.c:
        Fixed indentation
      storage/maria/ma_rt_split.c:
        Fixed indentation
        maria_data_on_page () -> _ma_get_page_used()
      storage/maria/ma_rt_test.c:
        Fixed indentation
      storage/maria/ma_search.c:
        Remove support of using -1 as 'last used index' to _ma_check_index()
        maria_data_on_page() -> _ma_get_page_used()
        maria_data_on_page() -> _ma_get_used_and_node()
        Use keypage_header
      storage/maria/ma_sort.c:
        Changed some buffers to size_t
        Changed rec_per_key_part to double
      storage/maria/ma_static.c:
        Removed NEAR
        Added maria_uuid
      storage/maria/ma_test2.c:
        Moevd testflag == 2 to correct place
        Remove test of reading with index number -1 (not supported anymore)
      storage/maria/ma_test_recovery.expected:
        Updated results
      storage/maria/ma_test_recovery:
        Changed tmp table names so that one can run maria_chk on them
      storage/maria/ma_write.c:
        Fixed indentation
        Use keypage_header
        Store index number on index pages
        maria_putint() -> _ma_store_page_used()
        maria_data_on_page() -> ma_get_used_and_node()
        maria_data_on_page() -> _ma_get_page_used()
        Added PAGE_CHECKSUM
        Added Maria handler to some functions
        Removed some not needed casts
      storage/maria/maria_chk.c:
        Added error handling for HA_ERR_NEW_FILE
        Added information about page checksums
        rec_per_key_part changed to double
        maria_data_on_page() -> _ma_get_page_used()
        Use keypage_header
      storage/maria/maria_def.h:
        Added IDENTICAL_PAGES_AFTER_RECOVERY and SANITY_CHECKS
        Changed rec_per_key_part to double
        Added nulls_per_key_part
        rec_per_key_rows -> records_at_analyze
        st_maria_info -> MARIA_HA
        Reserve place for new statistics variables, uuid, checksums per page etc.
        Removed NEAR tags
        Changed some prototypes to use my_bool and size_t
      storage/maria/maria_pack.c:
        st_maria_info -> MARIA_HA
        Fixed indentation
      storage/myisam/mi_dbug.c:
        Fix wrong debug output for ULONG
      mysys/my_rnd.c:
        New BitKeeper file ``mysys/my_rnd.c''
      mysys/my_uuid.c:
        New BitKeeper file ``mysys/my_uuid.c''
      496741d5
  10. 04 Oct, 2007 1 commit
  11. 02 Oct, 2007 1 commit
    • unknown's avatar
      WL#3072 - Maria recovery. · d0b9387b
      unknown authored
      * Recovery of the table's live checksum (CREATE TABLE ... CHECKSUM=1)
      is achieved in this patch. The table's live checksum
      (info->s->state.state.checksum) is updated in inwrite_rec_hook's
      under the log mutex when writing UNDO_ROW_INSERT|UPDATE|DELETE
      and REDO_DELETE_ALL. The checksum variation caused by the operation
      is stored in these UNDOs, so that the REDO phase, when it sees such
      UNDOs, can update the live checksum if it is older (state.is_of_lsn is
      lower) than the record. It is also used, as a nice add-on with no
      cost, to do less row checksum computation during the UNDO phase
      (as we have it in the record already).
      Doing this work, it became pressing to move in-write hooks
      (write_hook_for_redo() et al) to ma_blockrec.c.
      The 'parts' argument of inwrite_rec_hook is unpredictable (it comes
      mangled at this stage, for example by LSN compression) so it is
      replaced by a 'void* hook_arg', which is used to pass down information,
      currently only to write_hook_for_clr_end() (previous undo_lsn and
      type of undone record).
      * If from ha_maria, we print to stderr how many seconds (with one
      fractional digit) the REDO phase took, same for UNDO phase and for
      final table close. Just to give an indication for debugging and maybe
      also for Support.
      
      
      storage/maria/ha_maria.cc:
        question for Monty
      storage/maria/ma_blockrec.c:
        * log in-write hooks (write_hook_for_redo() etc) move from
        ma_loghandler.c to here; this is natural: the hooks are coupled
        to their callers (functions in ma_blockrec.c).
        * translog_write_record() now has a new argument "hook_arg";
        using it to pass down to write_hook_for_clr_end() the transaction's
        previous_undo_lsn and the type of the being undone record, and also
        to pass down to all UNDOs the live checksum variation caused by the
        operation.
        * If table has live checksum, store in UNDO_ROW_INSERT|UPDATE|DELETE
        and in CLR_END the checksum variation ("delta") caused by the
        operation. For example if a DELETE caused the table's live checksum
        to change from 123 to 456, we store in the UNDO_ROW_DELETE, in 4 bytes,
        the value 333 (456-123).
        * Instead of hard-coded "1" as length of the place where we store
        the undone record's type in CLR_END, use a symbol CLR_TYPE_STORE_SIZE;
        use macros clr_type_store and clr_type_korr.
        * write_block_record() has a new parameter 'old_record_checksum'
        which is the pre-computed checksum of old_record; that value is used
        to update the table's live checksum when writing UNDO_ROW_UPDATE|CLR_END.
        * In allocate_write_block_record(), if we are executing UNDO_ROW_DELETE
        the row's checksum is already computed.
        * _ma_update_block_record2() now expect the new row's checksum into
        cur_row.checksum (was already true) and the old row's checksum into
        new_row.checksum (that's new). Its two callers, maria_update() and
        _ma_apply_undo_row_update(), honour this.
        * When executing an UNDO_ROW_INSERT|UPDATE|DELETE in UNDO phase, pick
        up the checksum delta from the log record. It is then used to update
        the table's live checksum when writing CLR_END, and saves us a
        computation of record.
      storage/maria/ma_blockrec.h:
        in-write hooks move from ma_loghandler.c
      storage/maria/ma_check.c:
        more straightforward size of buffer
      storage/maria/ma_checkpoint.c:
        <= is enough
      storage/maria/ma_commit.c:
        new prototype of translog_write_record()
      storage/maria/ma_create.c:
        new prototype of translog_write_record()
      storage/maria/ma_delete.c:
        The row's checksum must be computed before calling(*delete_record)(),
        not after, because it must be known inside _ma_delete_block_record()
        (to update the table's live checksum when writing UNDO_ROW_DELETE).
        If deleting from a transactional table, live checksum was already updated
        when writing UNDO_ROW_DELETE.
      storage/maria/ma_delete_all.c:
        @todo is now done (in ma_loghandler.c)
      storage/maria/ma_delete_table.c:
        new prototype of translog_write_record()
      storage/maria/ma_loghandler.c:
        * in-write hooks move to ma_blockrec.c.
        * translog_write_record() gets a new argument 'hook_arg' which is
        passed down to pre|inwrite_rec_hook. It is more useful that 'parts'
        for those hooks, because when those hooks are called, 'parts' has
        possibly been mangled (like with LSN compression) and is so
        unpredictable.
        * fix for compiler warning (unused buffer_start when compiling without
        debug support)
        * Because checksum delta is stored into UNDO_ROW_INSERT|UPDATE|DELETE
        and CLR_END, but only if the table has live checksum, these records
        are not PSEUDOFIXEDLENGTH anymore, they are now VARIABLE_LENGTH (their
        length is X if no live checksum and X+4 otherwise).
        * add an inwrite_rec_hook for UNDO_ROW_UPDATE, which updates the
        table's live checksum. Update it also in hooks of UNDO_ROW_INSERT|
        DELETE and REDO_DELETE_ALL and CLR_END.
        * Bugfix: when reading a record in translog_read_record(), it happened
        that "length" became negative, because the function assumed that
        the record extended beyond the page's end, whereas it may be shorter.
      storage/maria/ma_loghandler.h:
        * Instead of hard-coded "1" and "4", use symbols and macros
        to store/retrieve the type of record which the CLR_END corresponds
        to, and the checksum variation caused by the operation which logs the
        record
        * translog_write_record() gets a new argument 'hook_arg' which is
        passed down to pre|inwrite_rec_hook. It is more useful that 'parts'
        for those hooks, because when those hooks are called, 'parts' has
        possibly been mangled (like with LSN compression) and is so
        unpredictable.
      storage/maria/ma_open.c:
        fix for "empty body in if() statement" (when compiling without safemutex)
      storage/maria/ma_pagecache.c:
        <= is enough
      storage/maria/ma_recovery.c:
        * print the time that each recovery phase (REDO/UNDO/flush) took;
        this is enabled only when recovering from ha_maria. Is it printed
        n seconds with a fractional part of one digit (like 123.4 seconds).
        * In the REDO phase, update the table's live checksum by using
        the checksum delta stored in UNDO_ROW_INSERT|DELETE|UPDATE and CLR_END.
        Update it too when seeing REDO_DELETE_ALL.
        * In the UNDO phase, when executing UNDO_ROW_INSERT, if the table does
        not have live checksum then reading the record's header (as done by
        the master loop of run_undo_phase()) is enough; otherwise we
        do a translog_read_record() to have the checksum delta ready
        for _ma_apply_undo_row_insert().
        * When at the end of the REDO phase we notice that there is an unfinished
        group of REDOs, don't assert in debug binaries, as I verified that it
        can happen in real life (with kill -9)
        * removing ' in #error as it confuses gcc3
      storage/maria/ma_rename.c:
        new prototype of translog_write_record()
      storage/maria/ma_test_recovery.expected:
        Change in output of ma_test_recovery: now all live checksums of
        original tables equal those of tables recreated by the REDO phase
        and those of tables fixed by the UNDO phase. I.e. recovery of
        the live checksum looks like working (which was after all the only
        goal of this changeset).
        I checked by hand that it's not just all live checksums which are
        now 0 and that's why they match. They are the old values like
        3757530372. maria.test has hard-coded checksum values in its result
        file so checks this too.
      storage/maria/ma_update.c:
        * It's useless to put up HA_STATE_CHANGED in 'key_changed',
        as we put up HA_STATE_CHANGED in info->update anyway.
        * We need to compute the old and new rows' checksum before calling
        (*update_record)(), as checksum delta must be known when logging
        UNDO_ROW_UPDATE which is done by _ma_update_block_record(). Note that
        some functions change the 'newrec' record (at least _ma_check_unique()
        does) so we cannot move the checksum computation too early in the
        function.
      storage/maria/ma_write.c:
        If inserting into a transactional table, live's checksum was
        already updated when writing UNDO_ROW_INSERT. The multiplication
        is a trick to save an if().
      storage/maria/unittest/ma_test_loghandler-t.c:
        new prototype of translog_write_record()
      storage/maria/unittest/ma_test_loghandler_first_lsn-t.c:
        new prototype of translog_write_record()
      storage/maria/unittest/ma_test_loghandler_max_lsn-t.c:
        new prototype of translog_write_record()
      storage/maria/unittest/ma_test_loghandler_multigroup-t.c:
        new prototype of translog_write_record()
      storage/maria/unittest/ma_test_loghandler_multithread-t.c:
        new prototype of translog_write_record()
      storage/maria/unittest/ma_test_loghandler_noflush-t.c:
        new prototype of translog_write_record()
      storage/maria/unittest/ma_test_loghandler_pagecache-t.c:
        new prototype of translog_write_record()
      storage/maria/unittest/ma_test_loghandler_purge-t.c:
        new prototype of translog_write_record()
      storage/myisam/sort.c:
        fix for compiler warnings in pushbuild (write_merge_key* functions
        didn't have their declaration match MARIA_HA::write_key).
      d0b9387b
  12. 12 Sep, 2007 1 commit
    • unknown's avatar
      WL#3071 Maria checkpoint · cec8ac3e
      unknown authored
      Finally this is the real checkpoint code.
      It however exhibits unstabilities when a checkpoint runs concurrently
      with data-modifying clients (table corruption, transaction log's
      assertions) so for now a checkpoint is taken only at startup after
      recovery and at shutdown, i.e. not in concurrent situations. Later
      we will let it run periodically, as well as flush dirty pages
      periodically (almost all needed code is there already, only pagecache
      code is written but not committed).
      WL#3072 Maria recovery
      * replacing UNDO_ROW_PURGE with CLR_END; testing of those CLR_END via
      ma_test2 which has INSERTs failing with duplicate keys.
      * replaying of REDO_RENAME_TABLE
      Now, off to test Recovery in ha_maria :)
      
      
      BitKeeper/deleted/.del-ma_least_recently_dirtied.c:
        Delete: storage/maria/ma_least_recently_dirtied.c
      BitKeeper/deleted/.del-ma_least_recently_dirtied.h:
        Delete: storage/maria/ma_least_recently_dirtied.h
      storage/maria/Makefile.am:
        compile Checkpoint module
      storage/maria/ha_maria.cc:
        When ha_maria starts, do a recovery from last checkpoint.
        Take a checkpoint when that recovery has ended and when ha_maria
        shuts down cleanly.
      storage/maria/ma_blockrec.c:
        * even if my_sync() fails we have to my_close() (otherwise we leak
        a descriptor)
        * UNDO_ROW_PURGE is replaced by a simple CLR_END for UNDO_ROW_INSERT,
        as promised in the old comment; it gives us skipping during the
        UNDO phase.
      storage/maria/ma_check.c:
        All REDOs before create_rename_lsn are ignored by Recovery. So
        create_rename_lsn must be set only after all data/index has been
        flushed and forced to disk. We thus move write_log_record_for_repair()
        to after _ma_flush_tables_files_after_repair().
      storage/maria/ma_checkpoint.c:
        Checkpoint module.
      storage/maria/ma_checkpoint.h:
        optional argument if caller wants a thread to periodically take
        checkpoints and flush dirty pages.
      storage/maria/ma_create.c:
        * no need to init some vars as the initial bzero(share) takes care of this.
        * update to new function's name
        * even if we fail in my_sync() we have to my_close()
      storage/maria/ma_extra.c:
        Checkpoint reads share->last_version under intern_lock, so we make
        maria_extra() update it under intern_lock. THR_LOCK_maria still needed
        because of _ma_test_if_reopen().
      storage/maria/ma_init.c:
        destroy checkpoint module when Maria shuts down.
      storage/maria/ma_loghandler.c:
        * UNDO_ROW_PURGE gone (see ma_blockrec.c)
        * we need to remember the LSN of the LOGREC_FILE_ID for a share,
        because this LSN is needed into the checkpoint record (Recovery wants
        to know the validity domain of an id->name mapping)
        * translog_get_horizon_no_lock() needed for Checkpoint
        * comment about failing assertion (Sanja knows)
        * translog_init_reader_data() thought that translog_read_record_header_scan()
        returns 0 in case of error, but 0 just means "0-length header".
        * translog_assign_id_to_share() now needs the MARIA_HA because
        LOGREC_FILE_ID uses a log-write hook.
        * Verify that (de)assignment of share->id happens only under intern_lock,
        as Checkpoint reads this id with intern_lock.
        * translog_purge() can accept TRANSLOG_ADDRESS, not necessarily
        a real LSN.
      storage/maria/ma_loghandler.h:
        prototype updates
      storage/maria/ma_open.c:
        no need to initialize "res"
      storage/maria/ma_pagecache.c:
        When taking a checkpoint, we don't need to know the maximum rec_lsn
        of dirty pages; this LSN was intended to be used in the two-checkpoint
        rule, but last_checkpoint_lsn is as good.
        4 bytes for stored_list_size is enough as PAGECACHE::blocks (number
        of blocks which the pagecache can contain) is int.
      storage/maria/ma_pagecache.h:
        new prototype
      storage/maria/ma_recovery.c:
        * added replaying of REDO_RENAME_TABLE
        * UNDO_ROW_PURGE gone (see ma_blockrec.c), replaced by CLR_END
        * Recovery from the last checkpoint record now possible
        * In new_table() we skip the table if the id->name mapping is older than
        create_rename_lsn (mapping dates from lsn_of_file_id).
        * in get_MARIA_HA_from_REDO_record() we skip the record
        if the id->name mapping is newer than the record (can happen if processing
        a record which is before the checkpoint record).
        * parse_checkpoint_record() has to return a LSN, that's what caller expects
      storage/maria/ma_rename.c:
        new function's name; log end zeroes of tables' names (ease recovery)
      storage/maria/ma_test2.c:
        * equivalent of ma_test1's --test-undo added (named -u here).
        * -t=1 now stops right after creating the table, so that
        we can test undoing of INSERTs with duplicate keys (which tests the
        CLR_END logged by _ma_write_abort_block_record()).
      storage/maria/ma_test_recovery.expected:
        Result of testing undoing of INSERTs with duplicate keys; there are
        some differences in maria_chk -dvv but they are normal (removing
        records does not shrink data/index file, does not put back the
        "analyzed, optimized keys"(etc) index state.
      storage/maria/ma_test_recovery:
        Test undoing of INSERTs with duplicate keys, using ma_test2;
        when such INSERT happens, it logs REDO_INSERT, UNDO_INSERT, REDO_DELETE,
        CLR_END; we abort after that, and test that CLR_END causes recovery
        to jump over UNDO_INSERT.
      storage/maria/ma_write.c:
        comment
      storage/maria/maria_chk.c:
        comment
      storage/maria/maria_def.h:
        * a new bit in MARIA_SHARE::in_checkpoint, used to build a list
        of unique shares during Checkpoint.
        * MARIA_SHARE::lsn_of_file_id added: the LSN of the last LOGREC_FILE_ID
        for this share; needed to know to which LSN domain the mappings
        found in the Checkpoint record apply (new mappings should not apply
        to old REDOs).
      storage/maria/trnman.c:
        * small changes to how trnman_collect_transactions() fills its buffer;
        it also uses a non-dummy lsn_read_non_atomic() found in ma_checkpoint.h
      cec8ac3e
  13. 10 Sep, 2007 1 commit
    • unknown's avatar
      Fixed some bugs when using undo of VARCHAR fields · 6aef814d
      unknown authored
      Fixed bug in undo_delete
      Fixed wrong error output from maria_check
      
      
      include/my_base.h:
        Added marker if we have null fields in table
      mysql-test/r/maria.result:
        checksum in maria now ignore null fields that are null
      sql/sql_table.cc:
        Ignore null fields that are now
        (Before enabling this, we have to change MyISAM to also skip null fields)
      storage/maria/ma_blockrec.c:
        More logging
        After merge fixes
        Fixed some bugs when using undo of VARCHAR fields
        Fixed bug in undo_delete (We can't use info->rec_buff here as this is used in write_block_record())
      storage/maria/ma_blockrec.h:
        ma_recordpos_to_dir_entry changed to return uint
      storage/maria/ma_check.c:
        Fixed wrong output in case of errors
      storage/maria/ma_create.c:
        Set share.base.pack_reclength more correct for block record
        Delete support for RAID
      storage/maria/ma_open.c:
        Don't calculate checksum fields with value NULL
      storage/maria/ma_test1.c:
        Fixed output from -v for VARCHAR keys
      storage/maria/ma_test_recovery.expected:
        Update results after adding new printf
        New checksums (because we now ignore nulls)
        Some file lengths are different, but think they are ok (didn't have time to investigate)
      storage/myisam/ha_myisam.cc:
        Fixed comment
      storage/myisam/mi_test1.c:
        Fixed bug
      6aef814d
  14. 09 Sep, 2007 1 commit
    • unknown's avatar
      Added applying of undo for updates · 155193a6
      unknown authored
      Fixed bug in duplicate key handling for block records during repair
      All read-row methods now return error number in case of error
      Don't calculate checksum for null fields
      Fixed bug when running maria_read_log with -o
      
      
      BUILD/SETUP.sh:
        Added STACK_DIRECTION
      BUILD/compile-pentium-debug-max:
        Moved STACK_DIRECTION to SETUP
      include/myisam.h:
        Added extra parameter to write_key
      storage/maria/ma_blockrec.c:
        Added applying of undo for updates
        Fixed indentation
        Removed some not needed casts
        Fixed wrong logging of CLR record
        Split ma_update_block_record to two functions to be able to reuse it from undo-applying
        Simplify filling of packed fields
        ma_record_block_record) now returns error number on failure
        Sligtly changed log record information for undo-update
      storage/maria/ma_check.c:
        Fixed bug in duplicate key handling for block records during repair
      storage/maria/ma_checksum.c:
        Don't calculate checksum for null fields
      storage/maria/ma_dynrec.c:
        _ma_read_dynamic_reocrd() now returns error number on error
        Rest of the changes are code simplification and indentation fixes
      storage/maria/ma_locking.c:
        Added comment
      storage/maria/ma_loghandler.c:
        More debugging
        Removed printing of total_record_length as this was always same as record_length
      storage/maria/ma_open.c:
        Allocate bitmap for changed fields
      storage/maria/ma_packrec.c:
        read_record now returns error number on error
      storage/maria/ma_recovery.c:
        Fixed wrong arguments to undo_row_update
      storage/maria/ma_statrec.c:
        read_record now returns error number on error (not 1)
        Code simplification
      storage/maria/ma_test1.c:
        Added exit possibility after update phase (to test undo of updates)
      storage/maria/maria_def.h:
        Include bitmap header file
      storage/maria/maria_read_log.c:
        Fixed bug when running with -o
      155193a6
  15. 07 Sep, 2007 2 commits
    • unknown's avatar
      If Maria engine is not compiled in, don't use page caches (fix · 64da2c3f
      unknown authored
      for compiler errors in pushbuild). Small bugfix.
      
      
      sql/handler.h:
        don't use pagecaches if no Maria
      storage/maria/ma_check.c:
        correcting mistake in previous push; need to call this function
        otherwise create_rename_lsn would not be updated at end of REPAIR.
      64da2c3f
    • unknown's avatar
      - WL#3072 Maria Recovery: · 2291f932
      unknown authored
      Recovery of state.records (the count of records which is stored into
      the header of the index file). For that, state.is_of_lsn is introduced;
      logic is explained in ma_recovery.c (look for "Recovery of the state").
      The net gain is that in case of crash, we now recover state.records,
      and it is idempotent (ma_test_recovery tests it).
      state.checksum is not recovered yet, mail sent for discussion.
      - WL#3071 Maria Checkpoint: preparation for it, by protecting
      all modifications of the state in memory or on disk with intern_lock
      (with the exception of the really-often-modified state.records,
      which is now protected with the log's lock, see ma_recovery.c
      (look for "Recovery of the state"). Also, if maria_close() sees that
      Checkpoint is looking at this table it will not my_free() the share.
      - don't compute row's checksum twice in case of UPDATE (correction
      to a bugfix I made yesterday).
      
      
      storage/maria/ha_maria.cc:
        protect state write with intern_lock (against Checkpoint)
      storage/maria/ma_blockrec.c:
        * don't reset trn->rec_lsn in _ma_unpin_all_pages(), because it
        should wait until we have corrected the allocation in the bitmap
        (as the REDO can serve to correct the allocation during Recovery);
        introducing _ma_finalize_row() for that.
        * In a changeset yesterday I moved computation of the checksum
        into write_block_record(), to fix a bug in UPDATE. Now I notice
        that maria_update() already computes the checksum, it's just that
        it puts it into info->cur_row while _ma_update_block_record()
        uses info->new_row; so, removing the checksum computation from
        write_block_record(), putting it back into allocate_and_write_block_record()
        (which is called only by INSERT and UNDO_DELETE), and copying
        cur_row->checksum into new_row->checksum in _ma_update_block_record().
      storage/maria/ma_check.c:
        new prototypes, they will take intern_lock when writing the state;
        also take intern_lock when changing share->kfile. In both cases
        this is to protect against Checkpoint reading/writing the state or reading
        kfile at the same time.
        Not updating create_rename_lsn directly at end of write_log_record_for_repair()
        as it wouldn't have intern_lock.
      storage/maria/ma_close.c:
        Checkpoint builds a list of shares (under THR_LOCK_maria), then it
        handles each such share (under intern_lock) (doing flushing etc);
        if maria_close() freed this share between the two, Checkpoint
        would see a bad pointer. To avoid this, when building the list Checkpoint
        marks each share, so that maria_close() knows it should not free it
        and Checkpoint will free it itself.
        Extending the zone covered by intern_lock to protect against
        Checkpoint reading kfile, writing state.
      storage/maria/ma_create.c:
        When we update create_rename_lsn, we also update is_of_lsn to
        the same value: it is logical, and allows us to test in maria_open()
        that the former is not bigger than the latter (the contrary is a sign
        of index header corruption, or severe logging bug which hinders
        Recovery, table needs a repair).
        _ma_update_create_rename_lsn_on_disk() also writes is_of_lsn;
        it now operates under intern_lock (protect against Checkpoint),
        a shortcut function is available for cases where acquiring
        intern_lock is not needed (table's creation or first open).
      storage/maria/ma_delete.c:
        if table is transactional, "records" is already decremented
        when logging UNDO_ROW_DELETE.
      storage/maria/ma_delete_all.c:
        comments
      storage/maria/ma_extra.c:
        Protect modifications of the state, in memory and/or on disk,
        with intern_lock, against a concurrent Checkpoint.
        When state goes to disk, update it's is_of_lsn (by calling
        the new _ma_state_info_write()).
        In HA_EXTRA_FORCE_REOPEN, don't set share->changed to 0 (undoing
        a change I made a few days ago) and ASK_MONTY
      storage/maria/ma_locking.c:
        no real code change here.
      storage/maria/ma_loghandler.c:
        Log-write-hooks for updating "state.records" under log's mutex
        when writing/updating/deleting a row or deleting all rows.
      storage/maria/ma_loghandler_lsn.h:
        merge (make LSN_ERROR and LSN_REPAIRED_BY_MARIA_CHK different)
      storage/maria/ma_open.c:
        When opening a table verify that is_of_lsn >= create_rename_lsn; if
        false the header must be corrupted.
        _ma_state_info_write() is split in two: _ma_state_info_write_sub()
        which is the old _ma_state_info_write(), and _ma_state_info_write()
        which additionally takes intern_lock if requested (to protect
        against Checkpoint) and updates is_of_lsn.
        _ma_open_keyfile() should change kfile.file under intern_lock
        to protect Checkpoint from reading a wrong kfile.file.
      storage/maria/ma_recovery.c:
        Recovery of state.records: when the REDO phase sees UNDO_ROW_INSERT
        which has a LSN > state.is_of_lsn it increments state.records.
        Same for UNDO_ROW_DELETE and UNDO_ROW_PURGE.
        When closing a table during Recovery, we know its state is at least
        as new as the current log record we are looking at, so increase
        is_of_lsn to the LSN of the current log record.
      storage/maria/ma_rename.c:
        update for new behaviour of _ma_update_create_rename_lsn_on_disk().
      storage/maria/ma_test1.c:
        update to new prototype
      storage/maria/ma_test2.c:
        update to new prototype (actually prototype was changed days ago,
        but compiler does not complain about the extra argument??)
      storage/maria/ma_test_recovery.expected:
        new result file of ma_test_recovery. Improvements: record
        count read from index's header is now always correct.
      storage/maria/ma_test_recovery:
        "rm" fails if file does not exist. Redirect stderr of script.
      storage/maria/ma_write.c:
        if table is transactional, "records" is already incremented when
        logging UNDO_ROW_INSERT. Comments.
      storage/maria/maria_chk.c:
        update is_of_lsn too
      storage/maria/maria_def.h:
        - MARIA_STATE_INFO::is_of_lsn which is used by Recovery. It is stored
        into the index file's header.
        - Checkpoint can now mark a table as "don't free this", and maria_close()
        can reply "ok then you will free it".
        - new functions
      storage/maria/maria_pack.c:
        update for new name
      2291f932
  16. 06 Sep, 2007 1 commit
    • unknown's avatar
      - speed optimization: · d5399185
      unknown authored
      minimize writes to transactional Maria tables: don't write
      data pages, state, and open_count at the end of each statement.
      Data pages will be written by a background thread periodically.
      State will be written by Checkpoint periodically.
      open_count serves to detect when a table is potentially damaged
      due to an unclean mysqld stop, but thanks to recovery an unclean
      mysqld stop will be corrected and so open_count becomes useless.
      As state is written less often, it is often obsolete on disk,
      we thus should avoid to read it from disk.
      - by removing the data page writes above, it is necessary to put
      it back at the start of some statements like check, repair and
      delete_all. It was already necessary in fact (see ma_delete_all.c).
      - disabling CACHE INDEX on Maria tables for now (fixes crash
      of test 'key_cache' when run with --default-storage-engine=maria).
      - correcting some fishy code in maria_extra.c (we possibly could lose
      index pages when doing a DROP TABLE under Windows, in theory).
      
      
      storage/maria/ha_maria.cc:
        disable CACHE INDEX in Maria for now (there is a single cache for now),
        it crashes and it's not a priority
      storage/maria/ma_bitmap.c:
        debug message
      storage/maria/ma_check.c:
        The statement before maria_repair() may not flush state,
        so it needs to be done by maria_repair() (indeed this function
        uses maria_open(HA_OPEN_COPY) so reads state from disk,
        so needs to find it up-to-date on disk).
        For safety (but normally this is not needed) we remove index blocks
        out of the cache before repairing.
        _ma_flush_blocks() becomes _ma_flush_table_files_after_repair():
        it now additionally flushes the data file and state and syncs files.
        As a side effect, the assertion "no WRITE_CACHE_USED" from
        _ma_flush_table_files() fired so we move all end_io_cache() done
        at the end of repair to before the calls to _ma_flush_table_files_after_repair().
      storage/maria/ma_close.c:
        when closing a transactional table, we fsync it. But we need to
        do this only after writing its state.
        We need to write the state at close time only for transactional
        tables (the other tables do that at last unlock).
        Putting back the O_RDONLY||crashed condition which I had
        removed earlier.
        Unmap the file before syncing it (does not matter now as Maria
        does not use mmap)
      storage/maria/ma_delete_all.c:
        need to flush data pages before chsize-ing it. Was needed even when
        we flushed data pages at the end of each statement, because we didn't
        anyway do it if under LOCK TABLES: the change here thus fixes this bug:
        create table t(a int) engine=maria;lock tables t write;
        insert into t values(1);delete from t;unlock tables;check table t;
        "Size of datafile is: 16384       Should be: 8192"
        (an obsolete page went to disk after the chsize(), at unlock time).
      storage/maria/ma_extra.c:
        When doing share->last_version=0, we make the MARIA_SHARE-in-memory
        invisible to future openers, so need to have an up-to-date state
        on disk for them. The same way, future openers will reopen the data
        and index file, so they will not find our cached blocks, so we
        need to flush them to disk.
        In HA_EXTRA_FORCE_REOPEN, this probably happens naturally as all
        tables normally get closed, we however add a safety flush.
        In HA_EXTRA_PREPARE_FOR_RENAME, we need to do the flushing. On
        Windows we additionally need to close files.
        In HA_EXTRA_PREPARE_FOR_DROP, we don't need to flush anything but
        remove dirty cached blocks from memory. On Windows we need to close
        files.
        Closing files forces us to sync them before (requirement for transactional
        tables).
        For mutex reasons (don't lock intern_lock twice), we move
        maria_lock_database() and _ma_decrement_open_count() first in the list
        of operations.
        Flush also data file in HA_EXTRA_FLUSH.
      storage/maria/ma_locking.c:
        For transactional tables:
          - don't write data pages / state at unlock time;
          as a consequence, "share->changed=0" cannot be done.
          - don't write state in _ma_writeinfo()
          - don't maintain open_count on disk (Recovery corrects the table in case of crash
          anyway, and we gain speed by not writing open_count to disk),
        For non-transactional tables, flush the state at unlock only
        if the table was changed (optimization).
        Code which read the state from disk is relevant only with
        external locking, we disable it (if want to re-enable it, it shouldn't
        for transactional tables as state on disk may be obsolete (such tables
        does not flush state at unlock anymore).
        The comment "We have to flush the write cache" is now wrong because
        maria_lock_database(F_UNLCK) now happens before thr_unlock(), and
        we are not using external locking.
      storage/maria/ma_open.c:
        _ma_state_info_read() is only used in ma_open.c, making it static
      storage/maria/ma_recovery.c:
        set MARIA_SHARE::changed to TRUE when we are going to apply a
        REDO/UNDO, so that the state gets flushed at close.
      storage/maria/ma_test_recovery.expected:
        Changes introduced by this patch:
        - good: the "open" (table open, not properly closed) is gone,
        it was pointless for a recovered table
        - bad: stemming from different moments of writing the index's state
        probably (_ma_writeinfo() used to write the state after every row
        write in ma_test* programs, doesn't anymore as the table is
        transactional): some differences in indexes (not relevant as we don't
        yet have recovery for them); some differences in count of records
        (changed from a wrong value to another wrong value) (not relevant
        as we don't recover this count correctly yet anyway, though
        a patch will be pushed soon).
      storage/maria/ma_test_recovery:
        for repeatable output, no names of varying directories.
      storage/maria/maria_chk.c:
        function renamed
      storage/maria/maria_def.h:
        Function became local to ma_open.c. Function renamed.
      d5399185
  17. 03 Sep, 2007 1 commit
    • unknown's avatar
      Fixed several bugs found by running *.test with maria engine · 5183a4b0
      unknown authored
      Renamed HA_EXTRA_PREAPRE_FOR_DELETE to HA_EXTRA_PERPARE_FOR_DROP
      Added HA_EXTRA_PREPARE_FOR_RENAME (as we in the code before used HA_EXTRA_PREPARE_FOR_DELETE also for renames which confused things)
      Allow multiple write locks for same page by same file handle
      Don't write table state if table is not changed
      
      
      include/my_base.h:
        Renamed HA_EXTRA_PREAPRE_FOR_DELETE to HA_EXTRA_PERPARE_FOR_DROP
        Added HA_EXTRA_PREPARE_FOR_RENAME (as we in the code before used HA_EXTRA_PREPARE_FOR_DELETE also for renames which confused things)
      mysql-test/r/maria.result:
        More tests of things that failed in other tests
      mysql-test/t/maria.test:
        More tests of things that failed in other tests
      sql/ha_partition.cc:
        HA_EXTRA_PREPARE_FOR_DELETE -> HA_EXTRA_PREPARE_FOR_DROP
        Use HA_EXTRA_PREPARE_FOR_RENAME for renames
      sql/ha_partition.h:
        HA_EXTRA_PREPARE_FOR_DELETE -> HA_EXTRA_PREPARE_FOR_DROP
        Use HA_EXTRA_PREPARE_FOR_RENAME for renames
      sql/lock.cc:
        Fixed comment
      sql/sql_table.cc:
        Fixed wrong usage of HA_EXTRA_PREAPRE_FOR_DELETE
      storage/maria/ha_maria.cc:
        Added missing _ma_renable_logging_for_table()  (When using with ALTER TABLE + repair index)
        Enabled fast generation of index
      storage/maria/ma_bitmap.c:
        Fixed bug when resetting full pages when page was a tail page
      storage/maria/ma_blockrec.c:
        Fixed several bugs found by running *.test with maria engine:
        During update we keep old changed pages locked with a write lock to be able to reuse them.
        - Fixed bug with allocated but not used tail part
        - Fixed bug with blob that only had tail part
        - Fixed bug when update reused a page (needed multiple write locks for same page)
        - Fixed bug when first extent was a tail block
      storage/maria/ma_check.c:
        Better error message when bitmap is destroyed
      storage/maria/ma_close.c:
        Only write status if file was changed.
        Fixed bug when maria_chk -e file_name changed the file.
      storage/maria/ma_dynrec.c:
        Removed not used argument to _ma_state_info_read_dsk
      storage/maria/ma_extra.c:
        HA_EXTRA_PREPARE_FOR_DELETE -> HA_EXTRA_PREPARE_FOR_DROP
        Use HA_EXTRA_PREPARE_FOR_RENAME for renames
        Only ignore flushing of pages for DROP (not rename)
      storage/maria/ma_locking.c:
        Removed not used argument to _ma_state_info_read_dsk
      storage/maria/ma_open.c:
        Removed not used argument to _ma_state_info_read_dsk
      storage/maria/ma_pagecache.c:
        Allow multiple write locks for same page by same file handle
        (Not yet complete, Sanja will fix)
      storage/maria/ma_recovery.c:
        HA_EXTRA_PREPARE_FOR_DELETE -> HA_EXTRA_PREPARE_FOR_DROP
      storage/maria/maria_def.h:
        Removed not used argument to _ma_state_info_read_dsk
      storage/myisam/mi_extra.c:
        HA_EXTRA_PREPARE_FOR_DELETE -> HA_EXTRA_PREPARE_FOR_DROP
        Use HA_EXTRA_PREPARE_FOR_RENAME for renames
        Only ignore flushing of pages for DROP (not rename)
      storage/myisammrg/ha_myisammrg.cc:
        HA_EXTRA_PREPARE_FOR_DELETE -> HA_EXTRA_PREPARE_FOR_DROP
        Use HA_EXTRA_PREPARE_FOR_RENAME for renames
      5183a4b0
  18. 29 Aug, 2007 1 commit
    • unknown's avatar
      WL#3072 Maria recovery · e27890ca
      unknown authored
      * create page cache before initializing engine and not after, because
      Maria's recovery needs a page cache
      * make the creation of a bitmap page more crash-resistent
      * bugfix (see ma_blockrec.c)
      * back to old way: create an 8k bitmap page when creating table
      * preparations for the UNDO phase: recreate TRNs
      * preparations for Checkpoint: list of dirty pages, testing
      of rec_lsn to know if page should be skipped during Recovery
      (unused in this patch as no Checkpoint module pushed yet)
      * maria_chk tags repaired table with a special LSN
      * reworking all around in ma_recovery.c (less duplication)
      
      
      mysys/my_realloc.c:
        noted an issue in my_realloc()
      sql/mysqld.cc:
        page cache needs to be created before engines are initialized,
        because Maria's initialization may do a recovery which needs
        the page cache.
      storage/maria/ha_maria.cc:
        update to new prototype
      storage/maria/ma_bitmap.c:
        when creating the first bitmap page we used chsize to 8192 bytes then 
        pwrite (overwrite) the last 2 bytes (8191-8192). If crash between
        the two operations, this leaves a bitmap page full without its end
        marker. A later recovery may try to read this page and find it
        exists and misses a marker and conclude it's corrupted and fail.
        Changing the chsize to only 8190 bytes: recovery will then find
        the page is too short and recreate it entirely.
      storage/maria/ma_blockrec.c:
        Fix for a bug: when executing a REDO, if the data page is created,
        data_file_length was increased before _ma_bitmap_set():
        _ma_bitmap_set() called _ma_read_bitmap_page() which, due to the
        increased data_file_length, expected to find a bitmap page on disk
        with a correct end marker; if the bitmap page didn't exist already
        in fact, this failed. Fixed by increasing data_file_length only after
        _ma_read_bitmap_page() has created the new bitmap page correctly.
        This bug could happen every time a REDO is about creating a new
        bitmap page.
      storage/maria/ma_check.c:
        empty data file has a bitmap page
      storage/maria/ma_control_file.c:
        useless parameter to ma_control_file_create_or_open(), just
        test if this is recovery.
      storage/maria/ma_control_file.h:
        new prototype
      storage/maria/ma_create.c:
        Back to how it was before: maria_create() creates an 8k bitmap page.
        Thus (bugfix) data_file_length needs to reflect this instead of being 0.
      storage/maria/ma_loghandler.c:
        as ma_test1 and ma_test2 now use real transactions and not
        dummy_transaction_object, REDO for INSERT/UPDATE/DELETE are always
        about real transactions, can assert this.
        A function for Recovery to assign a short id to a table.
      storage/maria/ma_loghandler.h:
        new function
      storage/maria/ma_loghandler_lsn.h:
        maria_chk tags repaired tables with this LSN
      storage/maria/ma_open.c:
        * enforce that DMLs on transactional tables use real transactions
        and not dummy_transaction_object.
        * test if table was repaired with maria_chk (which has to been
        seen as an import of an external table into the server), test
        validity of create_rename_lsn (header corruption detection)
        * comments.
      storage/maria/ma_recovery.c:
        * preparations for the UNDO phase: recreate TRNs
        * preparations for Checkpoint: list of dirty pages, testing
        of rec_lsn to know if page should be skipped during Recovery
        (unused in this patch as no Checkpoint module pushed yet)
        * reworking all around (less duplication)
      storage/maria/ma_recovery.h:
        a parameter to say if the UNDO phase should be skipped
      storage/maria/maria_chk.c:
        tag repaired tables with a special LSN
      storage/maria/maria_read_log.c:
        * update to new prototype
        * no UNDO phase in maria_read_log for now
      storage/maria/trnman.c:
        * a function for Recovery to create a transaction (TRN), needed
        in the UNDO phase
        * a function for Recovery to grab an existing transaction, needed
        in the UNDO phase (rollback all existing transactions)
      storage/maria/trnman_public.h:
        new functions
      e27890ca
  19. 09 Aug, 2007 2 commits
    • unknown's avatar
      * tests which use MERGE or INSERT DELAYED should run only · 0d301ee8
      unknown authored
      with engines which support that
      * temporarily adding option --global-subst to mysqltest so that
      the full testsuite can be run using Maria tables without failing
      on trivial differences (like diff in the engine clause of
      SHOW CREATE TABLE)
      * using recognizable tags for todos of the Maria team
      
      
      client/mysqltest.c:
        temporarily adding option --global-subst: its argument is X,Y.
        It replaces all occurrences of X by Y into mysqltest's result
        before the comparison with the expected result is done.
        This serves for when a test is run with --default-storage-engine=X
        where X is not MyISAM: tests using SHOW CREATE TABLE will always fail
        because SHOW CREATE TABLE prints X instead of MyISAM. With
        --global-subst=X,MyISAM , such trivial differences are eliminated and
        test may be reported as passing.
        For example, --global-subst=MARIA,MyISAM
        This is not good enough for merging into main trees! just for running
        many tests and finding bugs now!
      mysql-test/mysql-test-run.pl:
        new option --mysqltest to pass options to mysqltest (like we have
        --mysqld). Used for example like this:
        ./mtr --mysqltest=--global-subst=MARIA,MyISAM
      mysql-test/r/merge.result:
        update
      mysql-test/t/delayed.test:
        run test only with engines which support INSERT DELAYED
      mysql-test/t/merge.test:
        run test only with MyISAM tables, as they are required by MERGE
      sql/sql_delete.cc:
        recognizable tag
      sql/table.cc:
        recognizable tag
      storage/maria/ha_maria.cc:
        recognizable tag
      storage/maria/ma_check.c:
        recognizable tag
      storage/maria/ma_create.c:
        recognizable tag
      0d301ee8
    • unknown's avatar
      * "transactionality" needs to be preserved by TRUNCATE TABLE: · 95e2558f
      unknown authored
      a table with TRANSACTIONAL=x needs to still have it after TRUNCATE.
      No testcase, but without this fix, the frm and the Maria table
      got "out of sync" in this case:
      create table t1 (a int) row_format=page transactional=0;
      truncate table t1;
      After TRUNCATE, the Maria table (not the frm) was transactional
      (thus logging records, which is wrong).
      * fix for non-closed file at end of "maria_chk -r"
      
      
      sql/table.cc:
        "transactionality" needs to be preserved when truncating.
        It's behind a if() to not cancel the hack added to mysql_truncate()
        today for temporary Maria tables.
      storage/maria/ha_maria.cc:
        question for Monty (he also has a big mail from me on the same subject)
      storage/maria/ma_check.c:
        question for Monty (likely bugs)
      storage/maria/ma_create.c:
        debugging info
      storage/maria/ma_open.c:
        fix for datafile left open at end of "maria_chk -r":
        ma_open_datafile() happens after _ma_bitmap_init(), it sets dfile.file
        so needs to set share->bitmap.file.file too (they are copies of
        each other). Otherwise it breaks how closing of files works in
        BLOCK_RECORD (which is that info.dfile.file is not closed
        but share->bitmap.file.file is closed): not setting share->bitmap.file.file
        can lead to forgetting to close a file or closing a wrong file.
      95e2558f
  20. 08 Aug, 2007 3 commits
    • unknown's avatar
      as I disabled creation of the first empty bitmap at creation time, · 83ea6e4f
      unknown authored
      I disable it too for repair time.
      
      
      83ea6e4f
    • unknown's avatar
      Fix for · ddac4525
      unknown authored
      ./mtr --mysqld=--default-storage-engine=maria --mem ps:
      I got "can't sync on file UNOPENED" among the messages of REPAIR TABLE;
      due to a missing setting of bitmap.file.file to -1.
      Maria had two names "Maria" and "MARIA", using one now: "MARIA".
      
      
      storage/maria/ha_maria.cc:
        plug.in uses "MARIA". Some code apparently picks the name from
        plug.in (SHOW CREATE TABLE, run ps.test on Maria tables), other from
        mysql_declare_plugin (SHOW CREATE TABLE on partitioned tables, run
        partition.test with Maria tables), better make names identical.
      storage/maria/ma_check.c:
        running ps.test on Maria tables I got "can't sync on file UNOPENED"
        among the messages of REPAIR TABLE. That was due to maria_repair()
        closing the data file, setting info->dfile.file to -1 to prevent
        a wrong double close, but forgetting to also set
        info->s->bitmap.file.file to -1; it left it unchanged and so,
        when close_thread_tables() closed the old version of the repaired
        table, _ma_once_end_block_record() tried to fsync the closed
        descriptor, resulting in a message.
        Basically, when setting info->dfile.file to something it's always
        safe and recommended to set bitmap.file.file to the same value
        as it's just a copy of the same descriptor see _ma_bitmap_init().
        Using change_data_file_descriptor() for that.
        Changing that function to use MY_WME as it looks safe.
      storage/maria/ma_close.c:
        no need to make the index file durable if table is not transactional
      ddac4525
    • unknown's avatar
      propagation to maria_repair_by_sort() and maria_repair_parallel() · 62b30a19
      unknown authored
      of bugfix made to maria_repair() yesterday.
      Fail "bk delta" (and thus "bk citool") if an added or modified line
      of a C/C++ file has white space at end of line
      
      
      BitKeeper/triggers/pre-delta:
        detection gave false alarm on added newline
      storage/maria/ma_check.c:
        propagation to maria_repair_by_sort() and maria_repair_parallel()
        of bugfix made to maria_repair() yesterday. No effect now as
        those two repair variants are never used with BLOCK_RECORD.
      62b30a19
  21. 07 Aug, 2007 2 commits
    • unknown's avatar
      Fix for errors during: · 1ad3a05d
      unknown authored
      "./mtr --mysqld=--default-storage-engine=maria mysqldump".
      First problem was use of INSERT DELAYED and MERGE tables without
      specifying that the tables to create should always be MyISAM.
      After fixing this, no rows were returned by the final SELECT of the
      "BUG 19025" portion of the test. Simplified problem was:
      LOCK TABLES `t1` WRITE;
      /*!40000 ALTER TABLE `t1` DISABLE KEYS */;
      INSERT INTO `t1` VALUES ('bla',1000),('bla',1001),('bla',1002);
      /*!40000 ALTER TABLE `t1` ENABLE KEYS */;
      UNLOCK TABLES;
      select * from t1;
      The SELECT would find no rows. Reason: ENABLE KEYS does a maria_repair();
      but data pages are still in the page cache and not on disk (because
      they were not flushed because maria_lock_database(F_UNLCK) was
      not called at the end of INSERT because under LOCK TABLES).
      At start of maria_repair(), sort_info.filelength is set to the
      physical size of the data file (=> too small because pages are in
      cache and not on disk).
      Then in sort_get_next_record(), when seeing end-of-file, this is done:
      sort_param->max_pos= sort_info->filelength;
      Further in maria_repair(), this is done:
      info->state->data_file_length= sort_param.max_pos;
      and so data_file_length is smaller (0) than reality (16384).
      This makes SELECT think EOF is where it is not, and thus find
      no rows.
      This is fixed by flushing all data pages at the start of maria_repair()
      (no performance problem is introduced as in common cases where
      ALTER TABLE is not under LOCK TABLES, the previous statement did
      this flush anyway).
      Another reason to do this flush is that, if not doing it, old cached
      pages might go down onto the repaired data file at a later point
      and thus corrupt it (assume a REPAIR non-QUICK).
      A similar bug is fixed:
      LOCK TABLES WRITE; INSERT; CHECK TABLE;
      reports "Size of datafile is: 0         Should be: 16384"
      again because the physical size was read without a preliminary
      page cache flush.
      
      
      mysql-test/r/maria.result:
        result update
      mysql-test/r/mysqldump.result:
        result update
      mysql-test/t/maria.test:
        adding test for fixed bug in LOCK TABLES + CHECK TABLE + block format.
        Disabling portion which hits "incorrect key file" but still
        letting it make the test fail (Monty to fix).
      mysql-test/t/mysqldump.test:
        in places where test expects engine to support INSERT DELAYED and
        be includable in a MERGE table, i.e. be MyISAM, we explicitely
        ask for MyISAM.
      storage/maria/ma_check.c:
        Before reading the data file's physical size with my_seek(MY_SEEK_END)
        during maria_chk_size() and maria_repair(), we must flush this
        data file, otherwise physical size is misleading and leads to
        - CHECK TABLE finding the table corrupted ("size of datafile should be"
        error)
        - ALTER TABLE ENABLE KEYS losing rows (maria_repair()
        setting data_file_length to a too small value => later SELECT does
        not find rows though they are in the data file).
        This fixes the "mysqldump.test" failure.
        sort_info.filelength contains the physical size, re-using it.
      1ad3a05d
    • unknown's avatar
      Fix for three bugs: · 72c3c369
      unknown authored
      number 1: "./mtr --mysqld=--default-storage-engine=maria backup"
      restored no rows (forgot to flush data pages before my_copy(),
      and also the maria_repair() used by ha_maria::restore() needed
      a correct data_file_length to not miss rows). [note that BACKUP
      TABLE will be removed anyway in 5.2]
      number 2: "./mtr --mysqld=--default-storage-engine=maria bootstrap"
      caused segfault (uninitialized variable)
      number 3: "./mtr --mysqld=--default-storage-engine=maria check"
      showed warning in CHECK TABLE (maria_create() created a non-empty
      data file with data_file_length==0).
      
      
      storage/maria/ha_maria.cc:
        in ha_maria::backup, need to flush the data file before copying it,
        otherwise data misses from the copy (bug 1)
      storage/maria/ma_bitmap.c:
        when allocating data at the end of the bitmap, best_data is at "end",
        should not be left to 0 (bug 2)
      storage/maria/ma_check.c:
        _ma_scan_block_record() is used in QUICK repair. It relies on
        data_file_length. RESTORE TABLE mixes the MAI of an empty table
        (so, data_file_length==0) with an non-empty MAD, and does a 
        QUICK repair; that got fooled (thought it had hit EOF immediately,
        so found no records) (bug 1)
      storage/maria/ma_create.c:
        At the end of maria_create() we have, in the index file,
        data_file_length==0, while the data file has a bitmap page (8192).
        This inconsistency makes CHECK TABLE rightly complain.
        Fixed by not creating a first bitmap page during maria_create()
        (also saves disk space) (bug 3) Question for Monty.
      storage/maria/ma_extra.c:
        A function to flush the data and index files before one can
        use OS syscalls (reads, writes) on those files. For example,
        ha_maria::backup() does a my_copy() of the data file and so
        all cached pieces of this file must be sent to the OS (bug 1)
        This function will have to be used elsewhere in Maria, several places
        have not been updated when we added pagecache-ing of the data file
        (they still only flush the index file), they are probable bugs.
      storage/maria/maria_def.h:
        new function. Needs to be visible from ha_maria::backup.
      72c3c369
  22. 01 Aug, 2007 1 commit
    • unknown's avatar
      WL#3072 Maria recovery: · 372b4ed4
      unknown authored
      preparation to enable recoverability of the table's state, more exactly
      info->state->checksum. This will require that info->state->checksum
      be updated in a inwrite_hook when writing an UNDO record, thus
      info->cur_row.checksum needs to be accessible to inwrite_hook, so we
      make translog_write_record() accept a MARIA_HA* (info) instead of
      MARIA_SHARE* (info->s); with this, we will be able to access
      info->cur_row.checksum. Old code which needed the MARIA_SHARE
      can derive it from MARIA_HA. Fix for typos and compiler warnings.
      
      
      storage/maria/ma_blockrec.c:
        fix for new loghandler API. Removing strange lines (how could gcc
        accept that?)
      storage/maria/ma_check.c:
        fix for new loghandler API
      storage/maria/ma_delete_all.c:
        fix for new loghandler API
      storage/maria/ma_loghandler.c:
        functions now take a MARIA_HA in argument, this is more powerful
        than a MARIA_SHARE (MARIA_SHARE can be derived from MARIA_HA, not
        the other way around). MARIA_HA will be needed to allow recoverability
        of the table's state.
        Fixing wrong DBUG_PRINT ('i' is not the id).
        When writing the LOGREC_FILE_ID, we don't have a MARIA_HA around,
        so we cannot ask translog_write_record() to store the id for us;
        we thus store the file's id by ourselves. Alternative would have been
        to pass MARIA_HA to translog_assign_id_to_share() but I didn't like it.
      storage/maria/ma_loghandler.h:
        new loghandler API
      storage/maria/tablockman.c:
        fix for compiler warning (intptr is int on my machine)
      372b4ed4
  23. 27 Jul, 2007 1 commit
    • unknown's avatar
      merging MyISAM changes into Maria (not done in 5.1->maria merge of · a3d2ae46
      unknown authored
      Jul 7th). "maria.test" and "ps_maria.test" still fail;
      "ma_test_all" starts failing (MyISAM has the same issue see BUG#30094).
      
      
      include/maria.h:
        merging MyISAM changes into Maria
      mysys/mf_keycache.c:
        mi_test_all showed "floating point exception", this was already
        fixed in the latest 5.1, importing fix.
      sql/item_xmlfunc.cc:
        compiler warning (already fixed in latest 5.1)
      storage/maria/ha_maria.cc:
        merging MyISAM changes into Maria. See #ifdef ASK_MONTY.
      storage/maria/ha_maria.h:
        merging MyISAM changes into Maria
      storage/maria/ma_cache.c:
        merging MyISAM changes into Maria
      storage/maria/ma_check.c:
        merging MyISAM changes into Maria
      storage/maria/ma_create.c:
        merging MyISAM changes into Maria
      storage/maria/ma_dynrec.c:
        merging MyISAM changes into Maria
      storage/maria/ma_extra.c:
        merging MyISAM changes into Maria
      storage/maria/ma_ft_boolean_search.c:
        merging MyISAM changes into Maria
      storage/maria/ma_ft_nlq_search.c:
        merging MyISAM changes into Maria
      storage/maria/ma_info.c:
        merging MyISAM changes into Maria
      storage/maria/ma_key.c:
        merging MyISAM changes into Maria
      storage/maria/ma_loghandler.c:
        compiler warning (part->length is size_t)
      storage/maria/ma_open.c:
        merging MyISAM changes into Maria
      storage/maria/ma_preload.c:
        merging MyISAM changes into Maria
      storage/maria/ma_range.c:
        merging MyISAM changes into Maria
      storage/maria/ma_rkey.c:
        merging MyISAM changes into Maria
      storage/maria/ma_rt_index.c:
        merging MyISAM changes into Maria
      storage/maria/ma_rt_key.c:
        merging MyISAM changes into Maria
      storage/maria/ma_rt_split.c:
        merging MyISAM changes into Maria
      storage/maria/ma_search.c:
        merging MyISAM changes into Maria
      storage/maria/ma_sort.c:
        merging MyISAM changes into Maria
      storage/maria/maria_def.h:
        merging MyISAM changes into Maria
      a3d2ae46
  24. 04 Jul, 2007 2 commits
    • unknown's avatar
      Maria: correcting assertions (should be enforced only in multi-threaded · 3a1c7c91
      unknown authored
      mode) so ma_test_all works again; more error detection in ma_test_all;
      maria_control renamed to maria_log_control (Monty's suggestion,
      so that a "rm maria_log*" removes all log-related files). Disabling
      failing wrong assertion.
      
      
      storage/maria/ma_blockrec.c:
        disabling assertion which fails because cur_block is a
        local variable not initialized
      storage/maria/ma_check.c:
        comment
      storage/maria/ma_control_file.h:
        control file renamed
      storage/maria/ma_loghandler.c:
        assertions needed only in multi-threaded mode (ma_test1 and ma_test2
        are single-threaded, it's ok for them to use dummy_transaction_object
        with transactional tables: trn->rec_lsn can be set without interfering
        with other threads).
      storage/maria/ma_test_all.sh:
        got caught by failures in some ma_test1 runs, which I didn't see
        because ma_test_all returned 0 and I didn't scroll up in the window;
        now using "set -e" to avoid that. Also testing that
        we get the errors and warnings we expect.
      storage/maria/unittest/Makefile.am:
        maria_control renamed
      3a1c7c91
    • unknown's avatar
      Fixed bytes to uchar and gptr to uchar* · 6ed46be7
      unknown authored
      6ed46be7
  25. 03 Jul, 2007 1 commit
    • unknown's avatar
      Maria: · 38812255
      unknown authored
      * Don't modify share->base.born_transactional; now it is a value carved
      in stone at creation time. share->now_transactional is what can be
      modified: it starts at born_transactional, can become false during
      ALTER TABLE (when we want no logging), and restored later.
      * Not resetting create_rename_lsn to 0 during delete_all or repair.
      * when we temporarily disable transactionality, we also change
      the page type to PAGECACHE_PLAIN_PAGE: it bypasses some work in the
      page cache (optimization), and avoids assertions related to LSNs.
      * Disable INSERT DELAYED for transactional tables, because
      durability could not be guaranteed (insertion may even not happen)
      
      
      mysys/mf_keycache.c:
        comment
      storage/maria/ha_maria.cc:
        * a transactional table cannot do INSERT DELAYED
        * ha_maria::save_transactional not needed anymore, as now instead
        we don't modify MARIA_SHARE::MARIA_BASE_INFO::born_transactional
        (born_transactional plays the role of save_transactional), and modify
        MARIA_SHARE::now_transactional.
        * REPAIR_TABLE log record is now logged by maria_repair()
        * comment why we rely on born_transactional to know if we should
        skipping a transaction.
        * putting together two if()s which test for F_UNLCK
      storage/maria/ha_maria.h:
        ha_maria::save_transactional not needed anymore (moved to the C layer)
      storage/maria/ma_blockrec.c:
        * For the block record's code (writing/updating/deleting records),
        all that counts is now_transactional, not born_transactional.
        * As we now set the page type to PAGECACHE_PLAIN_PAGE for tables
        which have now_transactional==FALSE, pagecache will not expect
        a meaningful LSN for them in pagecache_unlock_by_link(), so
        we can pass it LSN_IMPOSSIBLE.
      storage/maria/ma_check.c:
        * writing LOGREC_REPAIR_TABLE moves from ha_maria::repair()
        to maria_repair(), sounds cleaner (less functions to export).
        * when opening a table during REPAIR, don't use the realpath-ed name,
        as this may fail if the table has symlinked files (maria_open()
        would try to find the data and index file in the directory
        of unique_file_name, it would fail if data and index files are in
        different dirs); use the unresolved name, open_file_name, which is
        the argument which was passed to the maria_open() which created 'info'.
      storage/maria/ma_close.c:
        assert that when a statement is done with a table, it cleans up
      storage/maria/ma_create.c:
        new name
      storage/maria/ma_delete_all.c:
        * using now_transactional
        * no reason to reset create_rename_lsn during delete_all (a bug);
        also no reason to do it during repair: it was put there because
        a positive create_rename_lsn caused a call to check_and_set_lsn()
        which asserted in DBUG_ASSERT(block->type == PAGECACHE_LSN_PAGE);
        first solution was to use LSN_IMPOSSIBLE in _ma_unpin_all_pages() if
        not transactional; but then in the case of ALTER TABLE, with
        transactionality temporarily disabled, it asserted in
        DBUG_ASSERT(LSN_VALID(lsn)) in pagecache_fwrite() (PAGECACHE_LSN_PAGE
        page with zero LSN - bad). The additional solution is to use
        PAGECACHE_PLAIN_PAGE when we disable transactionality temporarily: this
        avoids checks on the LSN, and also bypasses (optimization) the "flush
        log up to LSN" call when the pagecache flushes our page (in other
        words, no WAL needed).
      storage/maria/ma_delete_table.c:
        use now_transactional
      storage/maria/ma_locking.c:
        assert that when a statement is done with a table, it cleans up.
      storage/maria/ma_loghandler.c:
        * now_transactional should be used to test if we want a log record.
        * Assertions to make sure dummy_transaction_object is not spoilt
        by its many users.
      storage/maria/ma_open.c:
        base.transactional -> base.born_transactional
      storage/maria/ma_pagecache.c:
        missing name for page's type. Comment for future.
      storage/maria/ma_rename.c:
        use now_transactional
      storage/maria/maria_chk.c:
        use born_transactional
      storage/maria/maria_def.h:
        MARIA_BASE_INFO::transactional renamed to born_transactional.
        MARIA_SHARE::now_transactional introduced.
        _ma_repair_write_log_record() is made local to ma_check.c.
        Macros to temporarily disable, and re-enable, transactionality for a
        table.
      storage/maria/maria_read_log.c:
        assertions and using the new macros. Adding a forgotten resetting
        when we finally close all tables.
      38812255
  26. 02 Jul, 2007 1 commit
    • unknown's avatar
      Merged with mysql-5.1 main tree. · 631ecaab
      unknown authored
      BUILD/compile-pentium-debug-max:
        Added definition after macro was removed from main tree. This will
        be fixed back in main tree later.
      631ecaab
  27. 01 Jul, 2007 1 commit
    • unknown's avatar
      Fixed REPAIR/CHECK/ANALYZE TABLE for tables with new BLOCK-ROW format. · d6f2fda6
      unknown authored
      Fixed maria_chk to repair BLOCK-ROW tables.
      Added CREATE options ROW_FORMAT=PAGE & TRANSACTIONAL= 0|1
      More DBUG information in a lot of functions
      Some minor code cleanups
      Enable handler errors earlier for better clear text error messages at handler startup / standalone usage.
      Don't print NULL strings in my_create_with_symlink();  Fixes core dump when used with --debug
      
      
      include/maria.h:
        Added extra variables needed for REPAIR with BLOCK records
      include/my_base.h:
        Added argument for opening copy of maria table without a shared object
      include/my_handler.h:
        Prototypes for my_handler_error_register() & my_handler_error_unregister()
      include/pagecache.h:
        Added PAGECACHE_READ_UNKNOWN_PAGE
      mysql-test/include/ps_conv.inc:
        Enforce creation of table as MyISAM (to allow one to use --default-storage-engine)
      mysql-test/r/maria.result:
        Moved some things to maria-connect.test
        Updared results as REPAIR now works
        Added tests for creation option TRANSACTIONAL
      mysql-test/r/ps_2myisam.result:
        Enforce creation of table as MyISAM (to allow one to use --default-storage-engine)
      mysql-test/r/ps_3innodb.result:
        Enforce creation of table as MyISAM (to allow one to use --default-storage-engine)
      mysql-test/r/ps_4heap.result:
        Enforce creation of table as MyISAM (to allow one to use --default-storage-engine)
      mysql-test/r/ps_5merge.result:
        Enforce creation of table as MyISAM (to allow one to use --default-storage-engine)
      mysql-test/r/ps_7ndb.result:
        Enforce creation of table as MyISAM (to allow one to use --default-storage-engine)
      mysql-test/r/ps_maria.result:
        Enforce creation of table as MyISAM (to allow one to use --default-storage-engine)
      mysql-test/t/maria.test:
        Moved some things to maria-connect.test
        Updared results as REPAIR now works
        Added tests for creation option TRANSACTIONAL
      mysys/mf_iocache.c:
        More debugging
      mysys/mf_tempfile.c:
        Added missing close()
      mysys/my_error.c:
        init_glob_errs() is now done in my_init()
      mysys/my_handler.c:
        Added functions to initialize handler error messages
      mysys/my_init.c:
        Moevd init_glob_errs() here.
      mysys/my_open.c:
        More comments
        More debugging
        Code cleanup (join multiple code paths) and indentation fixes. No change in logic.
      mysys/my_symlink2.c:
        Don't print NULL strings
      sql/handler.cc:
        Added printing of PAGE row type
        Moved out initializing of handler errors to allow handler to give better error messages at startup
      sql/handler.h:
        ROW_TYPE_PAGES -> ROW_TYPE_PAGE
      sql/lex.h:
        Added 'PAGE' and 'TRANSACTIONAL'
      sql/mysqld.cc:
        Initialize handler error messages early to get better error messages from handler startup
      sql/sql_show.cc:
        ROW_TYPE_PAGES -> ROW_TYPE_PAGE
      sql/sql_table.cc:
        Removed not needed initializer
      sql/sql_yacc.yy:
        Added CREATE options ROW_FORMAT=PAGE and TRANSACTIONAL=[0|1]
      sql/table.cc:
        Store transactional flag in .frm
        More comments
      sql-bench/example:
        Better example
      sql/table.h:
        Added transactional table option
      storage/maria/ha_maria.cc:
        More debug information
        Enable REPAIR
        Detect usage of TRANSACTIONAL table option
      storage/maria/ma_bitmap.c:
        More comments (from Guilhem)
      storage/maria/ma_blockrec.c:
        SANITY_CHECK -> SANITY_CHECKS (fixed typo)
        Write out pages on delete even if there is no rows. (Fixed problem with REPAIR)
        Removed some ASSERTS to runtime checks (for better REPAIR)
        Fixed bug when scanning rows
        More DBUG information
      storage/maria/ma_check.c:
        Partial rewrite to allow REPAIR of BLOCK/PAGE format.
        Repair of BLOCK format rows is for now only done with 'maria_repair()' (= repair through key cache)
        The new logic to repair rows with BLOCK format is:
        
        - Create new, unrelated MARIA_HA of the table
        - Create new datafile and associate it with new handler
        - Reset all statistic information in new handler
        - Copy all data to new handler with normal write operations
        - Move state of new handler to old handler
        - Close new handler
        - Close data file in old handler
        - Rename old data file to new data file.
        - Reopen data file in old handler
      storage/maria/ma_close.c:
        REmoved not needed block
      storage/maria/ma_create.c:
        Swap arguments to _ma_initialize_data_file()
      storage/maria/ma_delete_all.c:
        Split maria_delete_all_rows() to two functions to allow REPAIR to easily reset all status information.
      storage/maria/ma_dynrec.c:
        Added checksum argument to _ma_rec_check (multi-thread fix)
      storage/maria/ma_info.c:
        Indentation fix
      storage/maria/ma_init.c:
        Register error message to get better error message on init and when using as standalone module.
      storage/maria/ma_loghandler.c:
        Fixed typo that disabled some error detection by valgrind
      storage/maria/ma_open.c:
        Added 'calc_check_checksum()'
        Don't log things during repair
        Added option HA_OPEN_COPY to allow one to open a Maria table with an independent share (required by REPAIR)
      storage/maria/ma_pagecache.c:
        Fixed some compiler warnings
        Added support for PAGECACHE_READ_UNKNOWN_PAGE (used for scanning file without knowing page types)
      storage/maria/ma_test_all.sh:
        More test of REPAIR
      storage/maria/ma_update.c:
        Optimized checksum code
      storage/maria/maria_chk.c:
        Use DBUG_SET_INITIAL() to get DBUG to work with --parallel-repair
        Ensure we always use maria_repair() for BLOCK format (for now)
        More DBUG information
      storage/maria/maria_def.h:
        For now, always run with more checkings (SANITY_CHECKS)
        Added share->calc_check_checksum to be used with REPAIR / CHECK table.
        Swaped arguments to _ma_initialize_data_file()
      storage/myisam/ft_stopwords.c:
        Added DBUG information
      mysql-test/r/maria-connect.result:
        New BitKeeper file ``mysql-test/r/maria-connect.result''
      mysql-test/t/maria-connect.test:
        New BitKeeper file ``mysql-test/t/maria-connect.test''
      d6f2fda6
  28. 28 Jun, 2007 1 commit
    • unknown's avatar
      WL#3239 "log CREATE TABLE in Maria": write the log record _before_ · 4b1fe65b
      unknown authored
      creating the data file, and sync this log, so that the table cannot be
      used if log record didn't reach disk. The same way, we force the log
      in DROP/RENAME TABLE. Also in REPAIR TABLE though logging in this
      case is not polished.
      Making DELETE FROM t <no WHERE> atomic: we log the record before
      starting the operation, and will finish this op at Recovery if needed.
      
      
      storage/maria/ma_check.c:
        comment. Force the log record for the log to have a complete history.
      storage/maria/ma_create.c:
        better conformance to the text of WL#3239 "log CREATE TABLE in Maria":
        write the log record before creating the data file. This ensures
        that the log can be applied to an old backup in all circumstances.
        errpos=2 was wrong.
      storage/maria/ma_delete_all.c:
        making DELETE FROM t <no WHERE> atomic: we log the record before
        starting the operation, and will finish the operation at Recovery
        if needed. Thus there is no need to force files to disk.
      storage/maria/ma_delete_table.c:
        forcing the log before dropping a table, so that the log has the
        entire history.
      storage/maria/ma_loghandler.c:
        LOGREC_REDO_DELETE_ALL needs to set trn's rec_lsn so that the log's
        low-water mark and Checkpoint retain this record until the 
        delete operation has finished.
      storage/maria/ma_rename.c:
        force the log before renaming a table, so that the log has a complete
        history.
      4b1fe65b
  29. 27 Jun, 2007 1 commit
    • unknown's avatar
      WL#3072 - Maria recovery · 0cf96a32
      unknown authored
      For this scenario: server crashes (could be because a table is
      corrupted) and Recovery repeatedly crashes on this table. User repairs
      it with maria_chk (as REPAIR TABLE is not possible), restarts the
      server, Recovery runs: for Recovery to not apply old REDOs to this
      repaired table (which would fail: rows have moved), maria_chk sets
      create_rename_lsn to the max value. Later when the server opens
      the table via ha_maria, it sets the LSN to the correct current value.
      
      
      storage/maria/ma_check.c:
        using helper function
      storage/maria/ma_create.c:
        A new helper function which stores the create_rename_lsn 
        into the table's header on disk when we cannot wait for this to happen
        naturally at a later _ma_state_info_write().
      storage/maria/ma_delete_all.c:
        using helper function; so log_data now can be FILEID_STORE_SIZE.
      storage/maria/ma_open.c:
        When opening a transactional table in the server, we discover
        if it has been repaired with maria_chk and if yes, give it a correct
        create_rename_lsn.
      storage/maria/ma_rename.c:
        using helper function
      storage/maria/maria_chk.c:
        By setting create_rename_lsn to the maximum possible LSN, maria_chk
        ensures that old REDOs are not applied to the new table it is
        going to produce.
      storage/maria/maria_def.h:
        new helper function
      0cf96a32
  30. 22 Jun, 2007 1 commit
    • unknown's avatar
      - WL#3239 "log CREATE TABLE in Maria" · 1a962591
      unknown authored
      - WL#3240 "log DROP TABLE in Maria"
      - similarly, log RENAME TABLE, REPAIR/OPTIMIZE TABLE, and
      DELETE no_WHERE_clause (== the DELETE which just truncates the files)
      - create_rename_lsn added to MARIA_SHARE's state
      - all these operations (except DROP TABLE) also update the table's
      create_rename_lsn, which is needed for the correctness of
      Recovery (see function comment of _ma_repair_write_log_record()
      in ma_check.c)
      - write a COMMIT record when transaction commits.
      - don't log REDOs/UNDOs if this is an internal temporary table
      like inside ALTER TABLE (I expect this to be a big win). There was
      already no logging for user-created "CREATE TEMPORARY" tables.
      - don't fsync files/directories if the table is not transactional
      - in translog_write_record(), autogenerate a 2-byte-id for the table
      and log the "id->name" pair (LOGREC_FILE_ID); log
      LOGREC_LONG_TRANSACTION_ID; automatically store
      the table's 2-byte-id in any log record.
      - preparations for Checkpoint: translog_get_horizon(); pausing Checkpoint
      when some dirty pages are unknown; capturing trn->rec_lsn,
      trn->first_undo_lsn for Checkpoint and log's low-water-mark computing.
      - assertions, comments.
      
      
      storage/maria/Makefile.am:
        more files to build
      storage/maria/ha_maria.cc:
        - logging a REPAIR log record if REPAIR/OPTIMIZE was successful.
        - ha_maria::data_file_type does not have to be set in every info()
        call, just do it once in open().
        - if caller said that transactionality can be disabled (like if
        caller is ALTER TABLE) i.e. thd->transaction.on==FALSE, then we
        temporarily disable transactionality of the table in external_lock();
        that will ensure that no REDOs/UNDOs are logged for this possibly
        massive write operation (they are not needed, as if any write fails,
        the table will be dropped). We re-enable in external_lock(F_UNLCK),
        which in ALTER TABLE happens before the tmp table replaces the original
        one (which is good, as thus the final table will have a REDO RENAME
        and a correct create_rename_lsn).
        - when we commit we also have to write a log record, so
        trnman_commit_trn() calls become ma_commit() calls
        - at end of engine's initialization, we are potentially entering a
        multi-threaded dangerous world (clients are going to be accepted)
        and so some assertions of mutex-owning become enforceable, for that
        we set maria_multi_threaded=TRUE (see ma_control_file.c)
      storage/maria/ha_maria.h:
        new member ha_maria::save_transactional (see also ha_maria.cc)
      storage/maria/ma_blockrec.c:
        - fixing comments according to discussion with Monty
        - if a table is transactional but temporarily non-transactional
        (like in ALTER TABLE), we need to give a sensible LSN to the pages
        (and, if we give 0, pagecache asserts).
        - translog_write_record() now takes care of storing the share's
        2-byte-id in the log record
      storage/maria/ma_blockrec.h:
        fixing comment according to discussion with Monty
      storage/maria/ma_check.c:
        When REPAIR/OPTIMIZE modify the data/index file, if this is a
        transactional table, they must sync it; if they remove files or rename
        files, they must sync the directory, so that everything is durable.
        This is just applying to REPAIR/OPTIMIZE the logic already implemented
        in CREATE/DROP/RENAME a few months ago.
        Adding a function to write a LOGREC_REPAIR_TABLE at end of
        REPAIR/OPTIMIZE (called only by ha_maria, not by maria_chk), and
        to update the table's create_rename_lsn.
      storage/maria/ma_close.c:
        fix for a future bug
      storage/maria/ma_control_file.c:
        ensuring that if Maria is running in multi-threaded mode, anybody
        wanting to write to the control file and update
        last_checkpoint_lsn/last_logno owns the log's lock.
      storage/maria/ma_control_file.h:
        see ma_control_file.c
      storage/maria/ma_create.c:
        when creating a table:
        - sync it and its directory only if this is a transactional table
        and there is a log (no point in syncing in maria_chk)
        - decouple the two uses of linkname/linkname_ptr (for index file and
        for data file) into more variables, as we need to know all links
        until the moment we write the LOGREC_CREATE_TABLE.
        - set share.data_file_type early so that _ma_initialize_data_file()
        knows it (Monty's bugfix so that a table always has at least a bitmap
        page when it is created; so data-file is not 0 bytes anymore).
        - log a LOGREC_CREATE_TABLE; it contains the bytes which we have
        just written to the index file's header. Update table's
        create_rename_lsn.
        - syncing of kfile had been bugified in a previous merge, correcting
        - syncing of dfile is now needed as it's not empty anymore
        - in _ma_initialize_data_file(), use share's block_size and not the
        global one. This is a gratuitous change, both variables are equal,
        just that I find it more future-proof to use share-bound variable
        rather than global one.
      storage/maria/ma_delete_all.c:
        log a LOGREC_DELETE_ALL record when doing ma_delete_all_rows();
        update create_rename_lsn then.
      storage/maria/ma_delete_table.c:
        - logging LOGREC_DROP_TABLE; knowing if this is needed, requires
        knowing if the table is transactional, which requires opening the
        table.
        - we need to sync directories only if the table is transactional
      storage/maria/ma_extra.c:
        questions
      storage/maria/ma_init.c:
        when maria_end() is called, engine is not multithreaded
      storage/maria/ma_loghandler.c:
        - translog_inited has to be visible to ma_create() (see how it is used
        in ma_create())
        - checkpoint record will be a single record, not three
        - no REDO for TRUNCATE (TRUNCATE calls ma_create() internally so will
        log a REDO_CREATE)
        - adding REDO for DELETE no_WHERE_clause (fast DELETE of all rows by
        truncating the files), REPAIR.
        - MY_WAIT_IF_FULL to wait&retry if a log write hits a full disk
        - in translog_write_record(), if MARIA_SHARE does not yet have a
        2-byte-id, generate one for it and log LOGREC_FILE_ID; automatically
        store this short id into log records.
        - in translog_write_record(), if transaction has not logged its
        long trid, log LOGREC_LONG_TRANSACTION_ID.
        - For Checkpoint, we need to know the current end-of-log: adding
        translog_get_horizon().
        - For Control File, adding an assertion that the thread owns the
        log's lock (control file is protected by this lock)
      storage/maria/ma_loghandler.h:
        Changes in log records (see ma_loghandler.c).
        new prototypes, new functions.
      storage/maria/ma_loghandler_lsn.h:
        adding a type LSN_WITH_FLAGS especially for TRN::first_undo_lsn,
        where the most significant byte is used for flags.
      storage/maria/ma_open.c:
        storing the create_rename_lsn in the index file's header (in the
        state, precisely) and retrieving it from there.
      storage/maria/ma_pagecache.c:
        - my set_if_bigger was wrong, correcting it
        - if the first_in_switch list is not empty, it means that
        changed_blocks misses some dirty pages, so Checkpoint cannot run and
        needs to wait. A variable missing_blocks_in_changed_list is added to
        tell that (should it be named missing_blocks_in_changed_blocks?)
        - pagecache_collect_changed_blocks_with_lsn() now also tells the
        minimum rec_lsn (needed for low-water mark computation).
      storage/maria/ma_pagecache.h:
        see ma_pagecache.c
      storage/maria/ma_panic.c:
        comment
      storage/maria/ma_range.c:
        comment
      storage/maria/ma_rename.c:
        - logging LOGREC_RENAME_TABLE; knowing if this is needed, requires
        knowing if the table is transactional, which requires opening the
        table.
        - update create_rename_lsn
        - we need to sync directories only if the table is transactional
      storage/maria/ma_static.c:
        comment
      storage/maria/ma_test_all.sh:
        - tip for Valgrind-ing ma_test_all
        - do "export maria_path=somepath" before calling ma_test_all,
        if you want to run ma_test_all out of storage/maria (useful
        to have parallel runs, like one normal and one Valgrind, they
        must not use the same tables so need to run in different directories)
      storage/maria/maria_def.h:
        - state now contains, in memory and on disk, the create_rename_lsn
        - share now contains a 2-byte-id
      storage/maria/trnman.c:
        preparations for Checkpoint: capture trn->rec_lsn, trn->first_undo_lsn;
        minimum first_undo_lsn needed to know log's low-water-mark
      storage/maria/trnman.h:
        using most significant byte of first_undo_lsn to hold miscellaneous
        flags, for now TRANSACTION_LOGGED_LONG_ID.
        dummy_transaction_object is already declared in ma_static.c.
      storage/maria/trnman_public.h:
        dummy_transaction_object was declared in all files including
        trnman_public.h, while in fact it's a single object.
        new prototype
      storage/maria/unittest/ma_test_loghandler-t.c:
        update for new prototype
      storage/maria/unittest/ma_test_loghandler_multigroup-t.c:
        update for new prototype
      storage/maria/unittest/ma_test_loghandler_multithread-t.c:
        update for new prototype
      storage/maria/unittest/ma_test_loghandler_pagecache-t.c:
        update for new prototype
      storage/maria/ma_commit.c:
        function which wraps:
        - writing a LOGREC_COMMIT record (==commit on disk)
        - calling trnman_commit_trn() (=commit in memory)
      storage/maria/ma_commit.h:
        new header file
      .tree-is-private:
        this file is now needed to keep our tree private (don't push it
        to public trees). When 5.1 is merged into mysql-maria, we can abandon
        our maria-specific post-commit trigger; .tree_is_private will take
        care of keeping commit mails private. Don't push this file to public
        trees.
      1a962591
  31. 09 Jun, 2007 1 commit
    • unknown's avatar
      Fixed compiler warnings · fdfb5148
      unknown authored
      Fixed bug in ma_dbug.c that gave valgrind warning (only relevant when using --debug)
      Fixed bug in blob logging (Fixes valgrind warning)
      maria_getint() -> maria_data_on_page()
      
      
      mysys/safemalloc.c:
        Added debug function to print out where a piece of memory was allocated
      sql/opt_range.cc:
        Remove DBUG_PRINT of unitailized memory
      storage/maria/ma_blockrec.c:
        Fixed bug in blob logging
      storage/maria/ma_check.c:
        Fixed compiler warning
      storage/maria/ma_dbug.c:
        Added missed end++; Caused usage of unitialized memory for nullable keys that was not NULL
      storage/maria/ma_delete.c:
        maria_getint() -> maria_data_on_page()
      storage/maria/ma_init.c:
        Added header file to get rid of warning
      storage/maria/ma_key.c:
        More debugging
      storage/maria/ma_loghandler.c:
        Removed some wrong ';' to get rid of compiler errors when compiling without debugging
        Indentation fixes
        Removed not needed 'break's
        Fixed some compiler warnings
        Added code to detect logging of unitialized memory
      storage/maria/ma_page.c:
        maria_getint() -> maria_data_on_page()
        Clear rest of index page before writing when used with valgrind
        (Fixes warning of writing pages with unitialized data)
      storage/maria/ma_range.c:
        maria_getint() -> maria_data_on_page()
      storage/maria/ma_rt_index.c:
        maria_getint() -> maria_data_on_page()
      storage/maria/ma_rt_index.h:
        maria_getint() -> maria_data_on_page()
      storage/maria/ma_rt_key.c:
        maria_getint() -> maria_data_on_page()
      storage/maria/ma_rt_split.c:
        maria_getint() -> maria_data_on_page()
      storage/maria/ma_search.c:
        maria_getint() -> maria_data_on_page()
      storage/maria/ma_test1.c:
        Fixed compiler warning
      storage/maria/ma_write.c:
        maria_getint() -> maria_data_on_page()
      storage/maria/maria_chk.c:
        maria_getint() -> maria_data_on_page()
      storage/maria/maria_def.h:
        maria_getint() -> maria_data_on_page()
      storage/maria/unittest/ma_pagecache_consist.c:
        Fixed compiler warning
      storage/maria/unittest/ma_pagecache_single.c:
        Fixed compiler warning
      storage/maria/unittest/ma_test_loghandler-t.c:
        Fixed compiler warning
      storage/maria/unittest/ma_test_loghandler_multigroup-t.c:
        Fixed compiler warning
      storage/maria/unittest/ma_test_loghandler_multithread-t.c:
        Fixed compiler warning
      storage/maria/unittest/ma_test_loghandler_pagecache-t.c:
        Fixed compiler warning
      storage/myisam/mi_dbug.c:
        Added missed end++; Caused usage of unitialized memory for nullable keys that was not NULL
      fdfb5148
  32. 04 Jun, 2007 1 commit
    • unknown's avatar
      Fixed that ma_test_all works with -T (simple transaction/logging support) · ffe437be
      unknown authored
      Some fixes from Sanja
      
      
      BitKeeper/etc/ignore:
        added storage/maria/maria_log.*
      include/pagecache.h:
        Always have enum PAGECACHE_EMPTY_PAGE available
        (Simpler code)
      storage/maria/ma_bitmap.c:
        Reset 'debugging' bitmap when creating new one (fixes valgrind warning)
      storage/maria/ma_blockrec.c:
        Removed duplicate (wrong) initialization
        Reset not initialized variable
      storage/maria/ma_check.c:
        Use right page type
        (Patch from Sanja)
      storage/maria/ma_init.c:
        Reset logging in maria_end()
        (Fixes memory leak)
      storage/maria/ma_loghandler.c:
        Add missing copyright header
        Added checking of duplicate calls or calls without init to translog_destroy()
        Don't lock mutex before destroying them (not needed as you can't use a destroyed mutex anyway)
      storage/maria/ma_pagecache.c:
        Added extra page type text
        Trivial indentation fixes
      storage/maria/ma_test1.c:
        Added transaction setup
        (Patch from Sanja)
      storage/maria/ma_test2.c:
        Added transaction setup
        (Patch from Sanja)
      ffe437be
  33. 29 May, 2007 1 commit
    • unknown's avatar
      This patch is a collection of patches from from Sanja, Sergei and Monty. · 8f39541e
      unknown authored
      Added logging and pinning of pages to block format.
      Integration of transaction manager, log handler.
      Better page cache intergration
      Split trnman.h into two files, so that we don't have to include my_atomic.h into C++ programs.
      Renaming of structures, more comments, more debugging etc.
      Fixed problem with small head block + long varchar.
      Added extra argument to delete_record() and update_record() (needed for UNDO logging)
      Small changes to interface of pagecache and log handler.
      Change initialization of log_record_type_descriptors to not be depending on enum order.
      Use array of LEX_STRING's to send data to log handler
      Added 'dummy' transaction option to MARIA_INFO so that we can always assume 'trn' exists.
      
      
      include/lf.h:
        Interface fixes
        Rename of structures
        (Patch from Sergei via Sanja)
      include/my_atomic.h:
        More comments
      include/my_global.h:
        Added MY_ERRPTR
      include/pagecache.h:
        Added undo LSN when unlocking pages
      mysql-test/r/maria.result:
        Updated results
      mysql-test/t/maria.test:
        Added autocommit around lock tables
        (Patch from Sanja)
      mysys/lf_alloc-pin.c:
        Post-review fixes, simple optimizations
        More comments
        Struct slot renames
        Check amount of memory on stack
        (Patch from Sergei)
      mysys/lf_dynarray.c:
        More comments
      mysys/lf_hash.c:
        More comments
        After review fixes
        (Patch from Sergei)
      storage/maria/ha_maria.cc:
        Split trnman.h into two files, so that we don't have to include my_atomic.h into the .cc program.
        (Temporary fix to avoid bug in gcc)
        Move out all deferencing of the transaction structure.
        Transaction manager integrated (Patch from Sergei)
      storage/maria/ha_maria.h:
        Added prototype for start_stmt()
      storage/maria/lockman.c:
        Function call rename
      storage/maria/ma_bitmap.c:
        Mark deleted pages free from page cache
      storage/maria/ma_blockrec.c:
        Offset -> rownr
        More debugging
        Fixed problem with small head block + long varchar
        Added logging of changed pages
        Added logging of undo (Including only loggging of changed fields in case of update)
        Added pinning/unpinning of all changed pages
        More comments
        Added free_full_pages() as the same code was used in several places.
        fill_rows_parts() renamed as fill_insert_undo_parts()
        offset -> rownr
        Added some optimization of not transactional tables
        _ma_update_block_record() has new parameter, as we need original row to do efficent undo for update
      storage/maria/ma_blockrec.h:
        Added ROW_EXTENTS_ON_STACK
        Changed prototype for update and delete of row
      storage/maria/ma_check.c:
        Added original row to delete_record() call
      storage/maria/ma_control_file.h:
        Added ifdefs for C++
      storage/maria/ma_delete.c:
        Added original row to delete_record() call
        (Needed for efficent undo logging)
      storage/maria/ma_dynrec.c:
        Added extra argument to delete_record() and update_record()
        Removed not used variable
      storage/maria/ma_init.c:
        Initialize log handler
      storage/maria/ma_loghandler.c:
        Removed not used variable
        Change initialization of log_record_type_descriptors to not be depending on enum order
        Use array of LEX_STRING's to send data to log handler
      storage/maria/ma_loghandler.h:
        New defines
        Use array of LEX_STRING's to send data to log handler
      storage/maria/ma_open.c:
        Added 'dummy' transaction option to MARIA_INFO so that we can always assume 'trn' exists.
        Store in MARIA_SHARE->page_type if pages will have up to date LSN's
      storage/maria/ma_pagecache.c:
        Don't decrease number of readers when using pagecache_write()/pagecache_read()
        In pagecache_write() decrement request count if page was left pinned
        Added pagecache_delete_pages()
        Removed some casts
        Make trace output consistent with rest of code
        Simplify calling of DBUG_ASSERT(0)
        Only update LSN if the LSN is bigger than what's already on the page
        Added LSN parameter pagecache_unpin_page(), pagecache_unpin(), and pagecache_unlock()
        (Part of patch from Sanja)
      storage/maria/ma_static.c:
        Added 'dummy' transaction option to MARIA_INFO so that we can always assume 'trn' exists.
        Added default page cache
      storage/maria/ma_statrec.c:
        Added extra argument to delete_record() and update_record()
      storage/maria/ma_test1.c:
        Added option -T for transactions
      storage/maria/ma_test2.c:
        Added option -T for transactions
      storage/maria/ma_test_all.sh:
        Test with transactions
      storage/maria/ma_update.c:
        Changed prototype for update of row
      storage/maria/maria_def.h:
        Changed prototype for update & delete of row as block records need to access the old row
        Store in MARIA_SHARE->page_type if pages will have up to date LSN's
        Added MARIA_MAX_TREE_LEVELS to allow us to calculate the number of possible pinned pages we may need.
        Removed not used 'empty_bits_buffer'
        Added pointer to transaction object
        Added array for pinned pages
        Added log_row_parts array for logging of field data.
        Added MARIA_PINNED_PAGE to store pinned pages
      storage/maria/trnman.c:
        Added accessor functions to transaction object
        Added missing DBUG_RETURN()
        More debugging
        More comments
        Changed // comment of code to #ifdef NOT_USED
        Transaction manager integrated.
        Post review fixes
        Part of patch originally from Sergei
      storage/maria/trnman.h:
        Split trnman.h into two files, so that we don't have to include my_atomic.h into the .cc program.
        (Temporary fix to avoid bug in gcc)
      storage/maria/unittest/ma_pagecache_single.c:
        Added missing argument
        Added SKIP_BIG_TESTS
        (Patch from Sanja)
      storage/maria/unittest/ma_test_loghandler-t.c:
        Test logging with new LEX_STRING parameter
        (Patch from Sanja)
      storage/maria/unittest/ma_test_loghandler_multigroup-t.c:
        Test logging with new LEX_STRING parameter
        (Patch from Sanja)
      storage/maria/unittest/ma_test_loghandler_multithread-t.c:
        Test logging with new LEX_STRING parameter
        (Patch from Sanja)
      storage/maria/unittest/ma_test_loghandler_pagecache-t.c:
        Test logging with new LEX_STRING parameter
        (Patch from Sanja)
      storage/maria/unittest/trnman-t.c:
        Stack overflow detection
        (Patch from Sergei)
      unittest/unit.pl:
        Command-line options --big and --verbose
        (Patch from Sergei)
      unittest/mytap/tap.c:
        Detect --big
        (Patch from Sergei)
      unittest/mytap/tap.h:
        Skip_big_tests and SKIP_BIG_TESTS
        (Patch from Sergei)
      storage/maria/trnman_public.h:
        New BitKeeper file ``storage/maria/trnman_public.h''
      8f39541e