An error occurred fetching the project authors.
  1. 07 Sep, 2007 5 commits
    • unknown's avatar
      enable --with-maria-storage-engine · 602e13cb
      unknown authored
      602e13cb
    • unknown's avatar
      Fix for pushbuild maria.test failure, where directory syncing failed at the · 02fd80ba
      unknown authored
      end of translog_flush() when datadir was in /dev/shm.
      
      
      storage/maria/ma_loghandler.c:
        directory syncing can fail on shared memory devices (/dev/shm on Linux
        in this case); see my_sync_dir().
      02fd80ba
    • unknown's avatar
      If Maria engine is not compiled in, don't use page caches (fix · 64da2c3f
      unknown authored
      for compiler errors in pushbuild). Small bugfix.
      
      
      sql/handler.h:
        don't use pagecaches if no Maria
      storage/maria/ma_check.c:
        correcting mistake in previous push; need to call this function
        otherwise create_rename_lsn would not be updated at end of REPAIR.
      64da2c3f
    • unknown's avatar
      WL#3072 - Maria Recovery · 69d7db77
      unknown authored
      At the end of recovery, we initialize the transaction manager's
      trid generator with the maximum trid seen during the REDO phase.
      This ensures that trids always grow (needed for versioning),
      even after a crash.
      This patch is only preparation, as ma_recover() is not called
      from ha_maria yet.
      
      
      storage/maria/ha_maria.cc:
        trnman_init() needs argument now (soon trnman_init() will rather
        be done via ma_recover() and thus it will not be 0)
      storage/maria/ma_recovery.c:
        During the REDO phase, remember the max long trid of transactions
        which we have seen (both in the checkpoint record and the
        LOGREC_LONG_TRANSACTION_ID records)
      storage/maria/ma_test1.c:
        trnman_init() needs argument now
      storage/maria/ma_test2.c:
        trnman_init() needs argument now
      storage/maria/trnman.c:
        new argument to trnman_init() so that caller can decide which
        value the generator of trids starts from.
      storage/maria/trnman_public.h:
        trnman_init() needs argument now
      storage/maria/unittest/trnman-t.c:
        trnman_init() needs argument now
      69d7db77
    • unknown's avatar
      - WL#3072 Maria Recovery: · 2291f932
      unknown authored
      Recovery of state.records (the count of records which is stored into
      the header of the index file). For that, state.is_of_lsn is introduced;
      logic is explained in ma_recovery.c (look for "Recovery of the state").
      The net gain is that in case of crash, we now recover state.records,
      and it is idempotent (ma_test_recovery tests it).
      state.checksum is not recovered yet, mail sent for discussion.
      - WL#3071 Maria Checkpoint: preparation for it, by protecting
      all modifications of the state in memory or on disk with intern_lock
      (with the exception of the really-often-modified state.records,
      which is now protected with the log's lock, see ma_recovery.c
      (look for "Recovery of the state"). Also, if maria_close() sees that
      Checkpoint is looking at this table it will not my_free() the share.
      - don't compute row's checksum twice in case of UPDATE (correction
      to a bugfix I made yesterday).
      
      
      storage/maria/ha_maria.cc:
        protect state write with intern_lock (against Checkpoint)
      storage/maria/ma_blockrec.c:
        * don't reset trn->rec_lsn in _ma_unpin_all_pages(), because it
        should wait until we have corrected the allocation in the bitmap
        (as the REDO can serve to correct the allocation during Recovery);
        introducing _ma_finalize_row() for that.
        * In a changeset yesterday I moved computation of the checksum
        into write_block_record(), to fix a bug in UPDATE. Now I notice
        that maria_update() already computes the checksum, it's just that
        it puts it into info->cur_row while _ma_update_block_record()
        uses info->new_row; so, removing the checksum computation from
        write_block_record(), putting it back into allocate_and_write_block_record()
        (which is called only by INSERT and UNDO_DELETE), and copying
        cur_row->checksum into new_row->checksum in _ma_update_block_record().
      storage/maria/ma_check.c:
        new prototypes, they will take intern_lock when writing the state;
        also take intern_lock when changing share->kfile. In both cases
        this is to protect against Checkpoint reading/writing the state or reading
        kfile at the same time.
        Not updating create_rename_lsn directly at end of write_log_record_for_repair()
        as it wouldn't have intern_lock.
      storage/maria/ma_close.c:
        Checkpoint builds a list of shares (under THR_LOCK_maria), then it
        handles each such share (under intern_lock) (doing flushing etc);
        if maria_close() freed this share between the two, Checkpoint
        would see a bad pointer. To avoid this, when building the list Checkpoint
        marks each share, so that maria_close() knows it should not free it
        and Checkpoint will free it itself.
        Extending the zone covered by intern_lock to protect against
        Checkpoint reading kfile, writing state.
      storage/maria/ma_create.c:
        When we update create_rename_lsn, we also update is_of_lsn to
        the same value: it is logical, and allows us to test in maria_open()
        that the former is not bigger than the latter (the contrary is a sign
        of index header corruption, or severe logging bug which hinders
        Recovery, table needs a repair).
        _ma_update_create_rename_lsn_on_disk() also writes is_of_lsn;
        it now operates under intern_lock (protect against Checkpoint),
        a shortcut function is available for cases where acquiring
        intern_lock is not needed (table's creation or first open).
      storage/maria/ma_delete.c:
        if table is transactional, "records" is already decremented
        when logging UNDO_ROW_DELETE.
      storage/maria/ma_delete_all.c:
        comments
      storage/maria/ma_extra.c:
        Protect modifications of the state, in memory and/or on disk,
        with intern_lock, against a concurrent Checkpoint.
        When state goes to disk, update it's is_of_lsn (by calling
        the new _ma_state_info_write()).
        In HA_EXTRA_FORCE_REOPEN, don't set share->changed to 0 (undoing
        a change I made a few days ago) and ASK_MONTY
      storage/maria/ma_locking.c:
        no real code change here.
      storage/maria/ma_loghandler.c:
        Log-write-hooks for updating "state.records" under log's mutex
        when writing/updating/deleting a row or deleting all rows.
      storage/maria/ma_loghandler_lsn.h:
        merge (make LSN_ERROR and LSN_REPAIRED_BY_MARIA_CHK different)
      storage/maria/ma_open.c:
        When opening a table verify that is_of_lsn >= create_rename_lsn; if
        false the header must be corrupted.
        _ma_state_info_write() is split in two: _ma_state_info_write_sub()
        which is the old _ma_state_info_write(), and _ma_state_info_write()
        which additionally takes intern_lock if requested (to protect
        against Checkpoint) and updates is_of_lsn.
        _ma_open_keyfile() should change kfile.file under intern_lock
        to protect Checkpoint from reading a wrong kfile.file.
      storage/maria/ma_recovery.c:
        Recovery of state.records: when the REDO phase sees UNDO_ROW_INSERT
        which has a LSN > state.is_of_lsn it increments state.records.
        Same for UNDO_ROW_DELETE and UNDO_ROW_PURGE.
        When closing a table during Recovery, we know its state is at least
        as new as the current log record we are looking at, so increase
        is_of_lsn to the LSN of the current log record.
      storage/maria/ma_rename.c:
        update for new behaviour of _ma_update_create_rename_lsn_on_disk().
      storage/maria/ma_test1.c:
        update to new prototype
      storage/maria/ma_test2.c:
        update to new prototype (actually prototype was changed days ago,
        but compiler does not complain about the extra argument??)
      storage/maria/ma_test_recovery.expected:
        new result file of ma_test_recovery. Improvements: record
        count read from index's header is now always correct.
      storage/maria/ma_test_recovery:
        "rm" fails if file does not exist. Redirect stderr of script.
      storage/maria/ma_write.c:
        if table is transactional, "records" is already incremented when
        logging UNDO_ROW_INSERT. Comments.
      storage/maria/maria_chk.c:
        update is_of_lsn too
      storage/maria/maria_def.h:
        - MARIA_STATE_INFO::is_of_lsn which is used by Recovery. It is stored
        into the index file's header.
        - Checkpoint can now mark a table as "don't free this", and maria_close()
        can reply "ok then you will free it".
        - new functions
      storage/maria/maria_pack.c:
        update for new name
      2291f932
  2. 06 Sep, 2007 2 commits
    • unknown's avatar
      - speed optimization: · d5399185
      unknown authored
      minimize writes to transactional Maria tables: don't write
      data pages, state, and open_count at the end of each statement.
      Data pages will be written by a background thread periodically.
      State will be written by Checkpoint periodically.
      open_count serves to detect when a table is potentially damaged
      due to an unclean mysqld stop, but thanks to recovery an unclean
      mysqld stop will be corrected and so open_count becomes useless.
      As state is written less often, it is often obsolete on disk,
      we thus should avoid to read it from disk.
      - by removing the data page writes above, it is necessary to put
      it back at the start of some statements like check, repair and
      delete_all. It was already necessary in fact (see ma_delete_all.c).
      - disabling CACHE INDEX on Maria tables for now (fixes crash
      of test 'key_cache' when run with --default-storage-engine=maria).
      - correcting some fishy code in maria_extra.c (we possibly could lose
      index pages when doing a DROP TABLE under Windows, in theory).
      
      
      storage/maria/ha_maria.cc:
        disable CACHE INDEX in Maria for now (there is a single cache for now),
        it crashes and it's not a priority
      storage/maria/ma_bitmap.c:
        debug message
      storage/maria/ma_check.c:
        The statement before maria_repair() may not flush state,
        so it needs to be done by maria_repair() (indeed this function
        uses maria_open(HA_OPEN_COPY) so reads state from disk,
        so needs to find it up-to-date on disk).
        For safety (but normally this is not needed) we remove index blocks
        out of the cache before repairing.
        _ma_flush_blocks() becomes _ma_flush_table_files_after_repair():
        it now additionally flushes the data file and state and syncs files.
        As a side effect, the assertion "no WRITE_CACHE_USED" from
        _ma_flush_table_files() fired so we move all end_io_cache() done
        at the end of repair to before the calls to _ma_flush_table_files_after_repair().
      storage/maria/ma_close.c:
        when closing a transactional table, we fsync it. But we need to
        do this only after writing its state.
        We need to write the state at close time only for transactional
        tables (the other tables do that at last unlock).
        Putting back the O_RDONLY||crashed condition which I had
        removed earlier.
        Unmap the file before syncing it (does not matter now as Maria
        does not use mmap)
      storage/maria/ma_delete_all.c:
        need to flush data pages before chsize-ing it. Was needed even when
        we flushed data pages at the end of each statement, because we didn't
        anyway do it if under LOCK TABLES: the change here thus fixes this bug:
        create table t(a int) engine=maria;lock tables t write;
        insert into t values(1);delete from t;unlock tables;check table t;
        "Size of datafile is: 16384       Should be: 8192"
        (an obsolete page went to disk after the chsize(), at unlock time).
      storage/maria/ma_extra.c:
        When doing share->last_version=0, we make the MARIA_SHARE-in-memory
        invisible to future openers, so need to have an up-to-date state
        on disk for them. The same way, future openers will reopen the data
        and index file, so they will not find our cached blocks, so we
        need to flush them to disk.
        In HA_EXTRA_FORCE_REOPEN, this probably happens naturally as all
        tables normally get closed, we however add a safety flush.
        In HA_EXTRA_PREPARE_FOR_RENAME, we need to do the flushing. On
        Windows we additionally need to close files.
        In HA_EXTRA_PREPARE_FOR_DROP, we don't need to flush anything but
        remove dirty cached blocks from memory. On Windows we need to close
        files.
        Closing files forces us to sync them before (requirement for transactional
        tables).
        For mutex reasons (don't lock intern_lock twice), we move
        maria_lock_database() and _ma_decrement_open_count() first in the list
        of operations.
        Flush also data file in HA_EXTRA_FLUSH.
      storage/maria/ma_locking.c:
        For transactional tables:
          - don't write data pages / state at unlock time;
          as a consequence, "share->changed=0" cannot be done.
          - don't write state in _ma_writeinfo()
          - don't maintain open_count on disk (Recovery corrects the table in case of crash
          anyway, and we gain speed by not writing open_count to disk),
        For non-transactional tables, flush the state at unlock only
        if the table was changed (optimization).
        Code which read the state from disk is relevant only with
        external locking, we disable it (if want to re-enable it, it shouldn't
        for transactional tables as state on disk may be obsolete (such tables
        does not flush state at unlock anymore).
        The comment "We have to flush the write cache" is now wrong because
        maria_lock_database(F_UNLCK) now happens before thr_unlock(), and
        we are not using external locking.
      storage/maria/ma_open.c:
        _ma_state_info_read() is only used in ma_open.c, making it static
      storage/maria/ma_recovery.c:
        set MARIA_SHARE::changed to TRUE when we are going to apply a
        REDO/UNDO, so that the state gets flushed at close.
      storage/maria/ma_test_recovery.expected:
        Changes introduced by this patch:
        - good: the "open" (table open, not properly closed) is gone,
        it was pointless for a recovered table
        - bad: stemming from different moments of writing the index's state
        probably (_ma_writeinfo() used to write the state after every row
        write in ma_test* programs, doesn't anymore as the table is
        transactional): some differences in indexes (not relevant as we don't
        yet have recovery for them); some differences in count of records
        (changed from a wrong value to another wrong value) (not relevant
        as we don't recover this count correctly yet anyway, though
        a patch will be pushed soon).
      storage/maria/ma_test_recovery:
        for repeatable output, no names of varying directories.
      storage/maria/maria_chk.c:
        function renamed
      storage/maria/maria_def.h:
        Function became local to ma_open.c. Function renamed.
      d5399185
    • unknown's avatar
      WL#3072 Maria Recovery · ac4ad9bd
      unknown authored
      misc fixes of execution of UNDOs in the UNDO phase:
      - into the CLR_END, store the LSN of the _previous_ UNDO (we debated
      what was best, so far we're going with "previous"; later we can change
      to "current" if needed), and store the type of record which is being
      undone (needed to know how to update state.records when we see the
      CLR_END during the REDO phase).
      - declaring all UNDOs and CLR_END as "compressed"
      - when executing an UNDO in the UNDO phase, state.records is updated
      as a hook when writing CLR_END (needed for "recovery of the state"),
      and so is trn->undo_lsn (needed for when we have checkpoints).
      - bugfix (execution of UNDO_ROW_DELETE didn't store the correct checksum
      into the re-inserted row, maria_chk -r thus threw the row away).
      - modifications of ma_test1: where to stop is now driven by --testflag;
      --test-undo just tells how to stop (flush data, flush log, nothing).
      - ma_test_recovery: testing of the UNDO phase, more testing of the
      REDO phase, identification of a bug.
      
      
      storage/maria/ma_blockrec.c:
        - bugfix: execution of UNDO_ROW_DELETE didn't store the correct
        checksum into the row (leading to "maria_chk -r" eliminating the
        re-inserted row, net effect was that rollback appeared to have
        rolled back no deletion). Reason was that write_block_record() used
        info->cur_row.checksum, while "row" can be != &info->cur_row
        (case of UNDO_ROW_DELETE). After fixing this, problems with
        _ma_update_block_record() appeared; indeed checksum was computed
        by  allocate_and_write_block_record() while _ma_update_block_record()
        directly calls write_block_record(). Solution is to compute checksum
        in write_block_record() instead.
        - when executing an UNDO, we now pass the LSN of the _previous_ UNDO
        to block_format functions. This LSN can be 0 (if the being-executed UNDO
        was the transaction's first UNDO), so "undo_lsn==0" cannot work
        anymore to indicate "this is not UNDO work". Using undo_lsn==LSN_ERROR
        instead (this is an impossible LSN).
        - store into CLR_END the type of log record which was undone
        (INSERT/UPDATE/DELETE); needed for Recovery to know if/how it has
        to update state.records if it sees this CLR_END in the REDO phase.
        - when writing the CLR_END in _ma_apply_undo_row_insert(),
        the place to store file's id is log_data+LSN_STORE_SIZE.
        - in _ma_apply_undo_row_insert(), the records-- is moved
        to a hook when writing the CLR_END (this way it is under log's mutex
        which is needed for "recovery of the state")
      storage/maria/ma_loghandler.c:
        - all UNDOs, and CLR_END, start with the LSN of another UNDO; so
        we can declare them "compressed".
        - write_hook_for_clr_end() to set trn->undo_lsn (to the previous
        UNDO's LSN) under log's lock (like UNDOs set trn->undo_lsn under log's
        lock), and also update, if appropriate, state.records.
        - reset share->id to 0 when deassigning; not useful for now but
        sounds logical.
      storage/maria/ma_recovery.c:
        - if no table is found for a REDO, it's not an error; for an UNDO, it is
        - in the REDO phase, when we see a CLR_END we must update trn->undo_lsn
        and sometimes state.records.
        - in the UNDO phase, when we execute an UNDO_ROW_INSERT:
          * update trn->undo_lsn only after executing the record
          * store the _previous_ undo_lsn into the CLR_END
        - at the end of the REDO phase, when we recreate TRN objects, they
        have already their long id in the log (either via a
        LOGREC_LONG_TRANSACTION_ID, or in a checkpoint record), don't write
        a new, useless LOGREC_LONG_TRANSACTION_ID for them.
      storage/maria/ma_test1.c:
        * where to stop execution is now driven by --testflag and not --test-undo
        (ma_test2 already has --testflag for the same purpose). This allows
        us to do a clean stop (with commit) at any point.
        * --test-undo=# tells how to abort (flush all pages (which implies
        flushing log) or only log or nothing); all such "ways of crashing"
        are tested in ma_test_recovery
      storage/maria/ma_test_recovery:
        * Testing execution of UNDOs, with and without BLOBs.
        * Testing idempotency of REDOs.
        * See @todo for a probable bug with BLOBs.
        * maria_chk -rq instead of -r, as with -q it nicely stops on any
        problem in the data file (like the checksum bug see comment of
        ma_blockrec.c).
        * Testing if log was written by UNDO phase (often expected),
        not written by REDO phase (always expected).
        * Less output on the screen, compares with expected output in the end.
        * some shell thingies like "set --" and $# are courtesy of
        Danny and Pekka.
      storage/maria/maria_read_log.c:
        when only displaying the records, don't do an UNDO phase
      storage/maria/ma_test_recovery.expected:
        This is the expected output of a great part of ma_test_recovery.
        ma_test_recovery compares its output to the expected output
        and tells if different.
        If we look at this file it mentions differences in checksum
        (normal, it's not recovered yet) and in records count
        (getting a correct records' count when recovery starts on an
        already existing table, like when testing rollback,
        is coded but not yet pushed).
      ac4ad9bd
  3. 04 Sep, 2007 6 commits
    • unknown's avatar
      Merge bk-internal.mysql.com:/home/bk/mysql-maria · 58ac5254
      unknown authored
      into  mysql.com:/home/my/mysql-maria
      
      
      58ac5254
    • unknown's avatar
      Added undo of deleted row · 044c4103
      unknown authored
      Added part of undo of update row
      Extended ma_test1 for recovery testing
      Some bug fixes
      
      
      storage/maria/ha_maria.cc:
        Ignore 'state.split' in case of block records
      storage/maria/ma_bitmap.c:
        Added return value for _ma_bitmap_find_place() for how much data we should put on head page
      storage/maria/ma_blockrec.c:
        Added undo of deleted row.
        - Added logging of CLR_END records in write_block_record()
        - Split ma_write_init_block_record() to two functions to get better code reuse
        - Added _ma_apply_undo_row_delete()
        - Added ma_get_length()
        
        Added 'empty' prototype for undo_row_update()
        
        Fixed bug when moving data withing a head/tail page.
        Fixed bug when reading a page with bigger LSN but of different type than was expected.
        Store undo_lsn first in CLR_END record
        
        Simplified some code by adding local variables.
        Changed log format for UNDO_ROW_DELETE to store total length of used blobs
      storage/maria/ma_blockrec.h:
        Added prototypes for undo code.
      storage/maria/ma_pagecache.c:
        Allow plain page to change to LSN page (needed in recovery to apply UNDO)
      storage/maria/ma_recovery.c:
        Added undo handling of UNDO_ROW_DELETE and UNDO_ROW_UPDATE
      storage/maria/ma_test1.c:
        Extended --test-undo option to allow us to die after insert or after delete.
        Fixed bug in printing key values when using -v
      storage/maria/maria_def.h:
        Moved some variables around to be getter alignment
        Added length_buff buffer to be used during undo handling
      044c4103
    • unknown's avatar
      Check of transaction log descriptor table consistance added. · 73b073ff
      unknown authored
      Small fixes made.
      
      
      storage/maria/ma_loghandler.c:
        Check of transaction log descriptor table consistance added.\
        Incorrect record description fixed.
        Compiler warning fixed.
      storage/maria/ma_loghandler.h:
        fixed ident.
      storage/maria/unittest/ma_test_loghandler-t.c:
        Suppressing of automatic record writing
      storage/maria/unittest/ma_test_loghandler_first_lsn-t.c:
        Suppressing of automatic record writing
      storage/maria/unittest/ma_test_loghandler_max_lsn-t.c:
        Suppressing of automatic record writing
      storage/maria/unittest/ma_test_loghandler_multigroup-t.c:
        Suppressing of automatic record writing
      storage/maria/unittest/ma_test_loghandler_multithread-t.c:
        Suppressing of automatic record writing
      storage/maria/unittest/ma_test_loghandler_noflush-t.c:
        Suppressing of automatic record writing
      storage/maria/unittest/ma_test_loghandler_pagecache-t.c:
        Suppressing of automatic record writing
      storage/maria/unittest/ma_test_loghandler_purge-t.c:
        Suppressing of automatic record writing
      73b073ff
    • unknown's avatar
      Merge bk-internal.mysql.com:/home/bk/mysql-maria · b4b0bf47
      unknown authored
      into  mysql.com:/home/my/mysql-maria
      
      
      storage/maria/ma_pagecache.c:
        Auto merged
      b4b0bf47
    • unknown's avatar
      Added UNDO handling of insert during recovery · 03437ea0
      unknown authored
      storage/maria/ma_blockrec.c:
        Added UNDO handling of insert during recovery
        To do this, I also had to add write locking of tail pages during undo phase (As we need to access the same page twice if extents are split over two pages)
        Another way to handle the undo of insert would be to store the extent information as part of the UNDO_INSERT block.
      storage/maria/ma_blockrec.h:
        Added new prototype
      storage/maria/ma_loghandler.c:
        Changed type of CLR_END (to avoid crash in log handler)
        Removed not used variable
      storage/maria/ma_loghandler.h:
        Added TRN argument to record_execute_in_undo_phase()
      storage/maria/ma_pagecache.c:
        Hack for undo phase of recovery.  During REDO we work with PLAIN pages, but UNDO works with LSN pages, which caused an abort when trying to access a cached page.
      storage/maria/ma_recovery.c:
        Added execution of UNDO_ROW_INSERT
      storage/maria/ma_test1.c:
        Added option --test-undo for testing recovery with undo
      storage/maria/maria_read_log.c:
        Added processing of undos
      03437ea0
    • unknown's avatar
      Spelling of comments fixed. · fe892036
      unknown authored
      fe892036
  4. 03 Sep, 2007 3 commits
    • unknown's avatar
    • unknown's avatar
      Merge bk-internal.mysql.com:/home/bk/mysql-maria · 572ce24f
      unknown authored
      into  mysql.com:/home/my/mysql-maria
      
      
      572ce24f
    • unknown's avatar
      Fixed several bugs found by running *.test with maria engine · 5183a4b0
      unknown authored
      Renamed HA_EXTRA_PREAPRE_FOR_DELETE to HA_EXTRA_PERPARE_FOR_DROP
      Added HA_EXTRA_PREPARE_FOR_RENAME (as we in the code before used HA_EXTRA_PREPARE_FOR_DELETE also for renames which confused things)
      Allow multiple write locks for same page by same file handle
      Don't write table state if table is not changed
      
      
      include/my_base.h:
        Renamed HA_EXTRA_PREAPRE_FOR_DELETE to HA_EXTRA_PERPARE_FOR_DROP
        Added HA_EXTRA_PREPARE_FOR_RENAME (as we in the code before used HA_EXTRA_PREPARE_FOR_DELETE also for renames which confused things)
      mysql-test/r/maria.result:
        More tests of things that failed in other tests
      mysql-test/t/maria.test:
        More tests of things that failed in other tests
      sql/ha_partition.cc:
        HA_EXTRA_PREPARE_FOR_DELETE -> HA_EXTRA_PREPARE_FOR_DROP
        Use HA_EXTRA_PREPARE_FOR_RENAME for renames
      sql/ha_partition.h:
        HA_EXTRA_PREPARE_FOR_DELETE -> HA_EXTRA_PREPARE_FOR_DROP
        Use HA_EXTRA_PREPARE_FOR_RENAME for renames
      sql/lock.cc:
        Fixed comment
      sql/sql_table.cc:
        Fixed wrong usage of HA_EXTRA_PREAPRE_FOR_DELETE
      storage/maria/ha_maria.cc:
        Added missing _ma_renable_logging_for_table()  (When using with ALTER TABLE + repair index)
        Enabled fast generation of index
      storage/maria/ma_bitmap.c:
        Fixed bug when resetting full pages when page was a tail page
      storage/maria/ma_blockrec.c:
        Fixed several bugs found by running *.test with maria engine:
        During update we keep old changed pages locked with a write lock to be able to reuse them.
        - Fixed bug with allocated but not used tail part
        - Fixed bug with blob that only had tail part
        - Fixed bug when update reused a page (needed multiple write locks for same page)
        - Fixed bug when first extent was a tail block
      storage/maria/ma_check.c:
        Better error message when bitmap is destroyed
      storage/maria/ma_close.c:
        Only write status if file was changed.
        Fixed bug when maria_chk -e file_name changed the file.
      storage/maria/ma_dynrec.c:
        Removed not used argument to _ma_state_info_read_dsk
      storage/maria/ma_extra.c:
        HA_EXTRA_PREPARE_FOR_DELETE -> HA_EXTRA_PREPARE_FOR_DROP
        Use HA_EXTRA_PREPARE_FOR_RENAME for renames
        Only ignore flushing of pages for DROP (not rename)
      storage/maria/ma_locking.c:
        Removed not used argument to _ma_state_info_read_dsk
      storage/maria/ma_open.c:
        Removed not used argument to _ma_state_info_read_dsk
      storage/maria/ma_pagecache.c:
        Allow multiple write locks for same page by same file handle
        (Not yet complete, Sanja will fix)
      storage/maria/ma_recovery.c:
        HA_EXTRA_PREPARE_FOR_DELETE -> HA_EXTRA_PREPARE_FOR_DROP
      storage/maria/maria_def.h:
        Removed not used argument to _ma_state_info_read_dsk
      storage/myisam/mi_extra.c:
        HA_EXTRA_PREPARE_FOR_DELETE -> HA_EXTRA_PREPARE_FOR_DROP
        Use HA_EXTRA_PREPARE_FOR_RENAME for renames
        Only ignore flushing of pages for DROP (not rename)
      storage/myisammrg/ha_myisammrg.cc:
        HA_EXTRA_PREPARE_FOR_DELETE -> HA_EXTRA_PREPARE_FOR_DROP
        Use HA_EXTRA_PREPARE_FOR_RENAME for renames
      5183a4b0
  5. 31 Aug, 2007 5 commits
    • unknown's avatar
      Merge desktop.sanja.is.com.ua:/home/bell/mysql/bk/mysql-maria · 0a2220b6
      unknown authored
      into  desktop.sanja.is.com.ua:/home/bell/mysql/bk/work-maria-purge
      
      
      storage/maria/ma_loghandler.c:
        Auto merged
      storage/maria/ma_loghandler.h:
        Auto merged
      0a2220b6
    • unknown's avatar
      Merge desktop.sanja.is.com.ua:/home/bell/mysql/bk/mysql-maria · bd02914a
      unknown authored
      into  desktop.sanja.is.com.ua:/home/bell/mysql/bk/work-maria-test
      
      
      bd02914a
    • unknown's avatar
      Merge bk-internal.mysql.com:/home/bk/mysql-maria · f5a8966d
      unknown authored
      into  mysql.com:/home/my/mysql-maria
      
      
      storage/maria/ma_blockrec.c:
        Auto merged
      storage/maria/ma_open.c:
        SCCS merged
      f5a8966d
    • unknown's avatar
      Generalized the way update and redo extends the size of a directory record. · 2cc2f3e7
      unknown authored
      storage/maria/ma_blockrec.c:
        Generalized the way update and redo extends the size of a directory record.
        This will (for now) ensure that data files are idenitical after normal run and after a apply-log run.
      storage/maria/ma_open.c:
        Disabled reservation of transid on rows (for now) as these are not yet used.
        (I had to disable this as otherwise update thougth rows had grown in size when they hadn't and we had thus different row sizes on update and redo, which caused different block information)
      storage/maria/ma_test1.c:
        Added comment
      storage/maria/ma_test2.c:
        Do commit on error/abort
      storage/maria/ma_test_all.sh:
        Some more testing (to cover a bug that was not found in previous runs)
      storage/maria/ma_test_recovery:
        More tests
      2cc2f3e7
    • unknown's avatar
      Fixed bug in log "in progress" marking. · 4b302966
      unknown authored
      storage/maria/ma_loghandler.c:
        Comments fixed.
        Fixed loop starting value.
      4b302966
  6. 29 Aug, 2007 5 commits
    • unknown's avatar
      WL#3072 Maria recovery · 90b63bf7
      unknown authored
      manual merge of ma_recovery.c (too big conflict to resolve in fmtool);
      the merged Monty's code allows correct replaying of REDO_PURGE_BLOCKS
      and was originally in
      monty@mysql.com/narttu.mysql.fi|ChangeSet|20070829060310|44058
      
      
      storage/maria/ma_recovery.c:
        * manually merging Monty's and Sanja's changes of the two last weeks
        to my massively modified version of this file. The merged Monty's
        code allows correct replaying of REDO_PURGE_BLOCKS and was originally
        in monty@mysql.com/narttu.mysql.fi|ChangeSet|20070829060310|44058 .
        * Setting the state to "STATE_CHANGED|etc" in Recovery is more
        logically done when we update the state in memory (for example
        records++).
      90b63bf7
    • unknown's avatar
      Merge gbichot@bk-internal.mysql.com:/home/bk/mysql-maria · b113fe74
      unknown authored
      into  gbichot4.local:/home/mysql_src/mysql-maria-for-undo-phase
      
      
      storage/maria/ha_maria.cc:
        Auto merged
      storage/maria/ma_blockrec.c:
        Auto merged
      storage/maria/ma_loghandler.c:
        Auto merged
      storage/maria/ma_loghandler.h:
        Auto merged
      storage/maria/ma_loghandler_lsn.h:
        Auto merged
      storage/maria/maria_chk.c:
        Auto merged
      storage/maria/maria_read_log.c:
        Auto merged
      b113fe74
    • unknown's avatar
      cleanups · 4201d278
      unknown authored
      storage/maria/ma_commit.c:
        theoretically unneeded, and could cause problems (when trnman_commit_trn()
        ends the TRN may have been recycled and be in use by another thread
        already, we cannot touch it).
      storage/maria/maria_def.h:
        just include the existing file
      4201d278
    • unknown's avatar
      WL#3072 Maria recovery · e27890ca
      unknown authored
      * create page cache before initializing engine and not after, because
      Maria's recovery needs a page cache
      * make the creation of a bitmap page more crash-resistent
      * bugfix (see ma_blockrec.c)
      * back to old way: create an 8k bitmap page when creating table
      * preparations for the UNDO phase: recreate TRNs
      * preparations for Checkpoint: list of dirty pages, testing
      of rec_lsn to know if page should be skipped during Recovery
      (unused in this patch as no Checkpoint module pushed yet)
      * maria_chk tags repaired table with a special LSN
      * reworking all around in ma_recovery.c (less duplication)
      
      
      mysys/my_realloc.c:
        noted an issue in my_realloc()
      sql/mysqld.cc:
        page cache needs to be created before engines are initialized,
        because Maria's initialization may do a recovery which needs
        the page cache.
      storage/maria/ha_maria.cc:
        update to new prototype
      storage/maria/ma_bitmap.c:
        when creating the first bitmap page we used chsize to 8192 bytes then 
        pwrite (overwrite) the last 2 bytes (8191-8192). If crash between
        the two operations, this leaves a bitmap page full without its end
        marker. A later recovery may try to read this page and find it
        exists and misses a marker and conclude it's corrupted and fail.
        Changing the chsize to only 8190 bytes: recovery will then find
        the page is too short and recreate it entirely.
      storage/maria/ma_blockrec.c:
        Fix for a bug: when executing a REDO, if the data page is created,
        data_file_length was increased before _ma_bitmap_set():
        _ma_bitmap_set() called _ma_read_bitmap_page() which, due to the
        increased data_file_length, expected to find a bitmap page on disk
        with a correct end marker; if the bitmap page didn't exist already
        in fact, this failed. Fixed by increasing data_file_length only after
        _ma_read_bitmap_page() has created the new bitmap page correctly.
        This bug could happen every time a REDO is about creating a new
        bitmap page.
      storage/maria/ma_check.c:
        empty data file has a bitmap page
      storage/maria/ma_control_file.c:
        useless parameter to ma_control_file_create_or_open(), just
        test if this is recovery.
      storage/maria/ma_control_file.h:
        new prototype
      storage/maria/ma_create.c:
        Back to how it was before: maria_create() creates an 8k bitmap page.
        Thus (bugfix) data_file_length needs to reflect this instead of being 0.
      storage/maria/ma_loghandler.c:
        as ma_test1 and ma_test2 now use real transactions and not
        dummy_transaction_object, REDO for INSERT/UPDATE/DELETE are always
        about real transactions, can assert this.
        A function for Recovery to assign a short id to a table.
      storage/maria/ma_loghandler.h:
        new function
      storage/maria/ma_loghandler_lsn.h:
        maria_chk tags repaired tables with this LSN
      storage/maria/ma_open.c:
        * enforce that DMLs on transactional tables use real transactions
        and not dummy_transaction_object.
        * test if table was repaired with maria_chk (which has to been
        seen as an import of an external table into the server), test
        validity of create_rename_lsn (header corruption detection)
        * comments.
      storage/maria/ma_recovery.c:
        * preparations for the UNDO phase: recreate TRNs
        * preparations for Checkpoint: list of dirty pages, testing
        of rec_lsn to know if page should be skipped during Recovery
        (unused in this patch as no Checkpoint module pushed yet)
        * reworking all around (less duplication)
      storage/maria/ma_recovery.h:
        a parameter to say if the UNDO phase should be skipped
      storage/maria/maria_chk.c:
        tag repaired tables with a special LSN
      storage/maria/maria_read_log.c:
        * update to new prototype
        * no UNDO phase in maria_read_log for now
      storage/maria/trnman.c:
        * a function for Recovery to create a transaction (TRN), needed
        in the UNDO phase
        * a function for Recovery to grab an existing transaction, needed
        in the UNDO phase (rollback all existing transactions)
      storage/maria/trnman_public.h:
        new functions
      e27890ca
    • unknown's avatar
      Added maria_commit() and maria_begin() to be used with external tests · f7b766c0
      unknown authored
      Now ma_test1 -M -T and ma_test2 -M -T produces readable, applyable logs
      
      Note: The .MAD file is not binary identical after applying redo compare to a an original file.
      (This is becasue we don't have full information which function called PURGE_REDO_BLOCKS).
      To verify if a file was correctly applied, we now instead compare row checksums
      
      
      BitKeeper/etc/ignore:
        added storage/maria/tmp/*
      include/maria.h:
        Added maria_commit() and maria_begin() to be used with external tests
      storage/maria/ha_maria.cc:
        Ensure maria_def. is read in C mode
      storage/maria/ma_blockrec.c:
        Fixed redo handling.
        _ma_apply_redo_purge_blocks() updated to handle any number of purged blocks
        Removed code to make data file idenitcal after redo (can't easily be done). See changeset comments
        
        Now ma_test1 -M -T and ma_test2 -M -T produces readable, applyable logs
      storage/maria/ma_commit.c:
        More DBUG statements
        Moved variable declaration to start of function (portability fix)
        Added helper functions 'maria_commit()' and 'maria_begin()'
      storage/maria/ma_loghandler.c:
        Fixed wrong REDO_PURGE_BLOCKS initialization
      storage/maria/ma_recovery.c:
        Added UNDO_ROW_UPDATE
        Removed wrong setting of lsn (there was no lsn at the used position)
        Fixed REDO_PURGE_BLOCKS to handle any number of blocks
      storage/maria/ma_test1.c:
        Added transaction support (via maria_begin() & maria_commit()) to get a log that can be applied with maria_read_log
      storage/maria/ma_test2.c:
        Added transaction support (via maria_begin() & maria_commit()) to get a log that can be applied with maria_read_log
      storage/maria/ma_test_recovery:
        Create temporary files in maria/tmp
        Verify files with checksums instead of byte comparisons
      storage/maria/maria_chk.c:
        When using with -dss we only get  filename, records and checksum.
        This is useful to do a quick comparision if a files is identical to another one.
      storage/maria/maria_def.h:
        Added ma_commit()
      storage/maria/maria_read_log.c:
        Added --help
      f7b766c0
  7. 27 Aug, 2007 1 commit
    • unknown's avatar
      Very simple log files purger added. · 7c273b82
      unknown authored
      File number types fixed.
      
      
      storage/maria/unittest/Makefile.am:
        Test of purger added.
      storage/maria/unittest/ma_test_loghandler_purge-t.c:
        New BitKeeper file ``storage/maria/unittest/ma_test_loghandler_purge-t.c''
      7c273b82
  8. 22 Aug, 2007 1 commit
    • unknown's avatar
      Fixed compiler warnings · d430e5bf
      unknown authored
      Fixed wrong hash function prototype (causes failure on 64 bit systems)
      
      
      mysql-test/r/rpl_events.result:
        Removed wrong merge (result file is now identical as in 5.1 tree)
      mysys/lf_hash.c:
        Fixed compiler warning
      mysys/my_safehash.c:
        Fixed wrong hash function prototype (causes failure on 64 bit systems)
      storage/maria/unittest/ma_test_loghandler_max_lsn-t.c:
        Fixed compiler warning
      d430e5bf
  9. 21 Aug, 2007 2 commits
    • unknown's avatar
      Merge bk-internal.mysql.com:/home/bk/mysql-maria · a1f80c1e
      unknown authored
      into  mysql.com:/home/my/mysql-maria
      
      
      a1f80c1e
    • unknown's avatar
      Fixes for bugs found by maria.test and event*tests: · a8d94b4a
      unknown authored
      Fixed bug when doing rnd_read followed by update.
      Don't duplicate error messages in log
      Initialize transaction object properly. Fixed failure in event*tests when running with maria
      
      
      mysql-test/mysql-test-run.pl:
        Removed warning when running with --external
      mysql-test/r/maria.result:
        Added back disabled test
        Verified that the result changes are correct
      mysql-test/t/maria.test:
        Added back disabled test
      sql/handler.cc:
        More debugging.
        Simple style change
      sql/sql_class.cc:
        Initialize transaction object properly. Fixed failure in event*tests when running with maria
      storage/maria/ha_maria.cc:
        More dbug info
      storage/maria/ma_blockrec.c:
        Removed not needed line
      storage/maria/ma_rrnd.c:
        Removed not used code
        Ensure that cur_row.lastpos is always set when reading record with rnd. (Fixes failure in maria.test)
      storage/maria/maria_def.h:
        Don't call maria_print_error() except with EXTRA_DEBUG
        (Removes duplicate error messages when somethings goes wrong)
      a8d94b4a
  10. 19 Aug, 2007 1 commit
    • unknown's avatar
      Storing/getting maximum LSN of the record which parts written · d0398f72
      unknown authored
      in the file to the file header.
      
      
      storage/maria/ma_loghandler.h:
        Getting maximum LSN of the record which parts written
        in the file to the file header.
      storage/maria/unittest/Makefile.am:
        Test suite for getting max LSN added.
      storage/maria/unittest/ma_test_loghandler_first_lsn-t.c:
        Spelling fixed.
        Cleanup fixed.
      storage/maria/unittest/ma_test_loghandler_noflush-t.c:
        Cleanup fixed.
      storage/maria/unittest/ma_test_loghandler_max_lsn-t.c:
        New BitKeeper file ``storage/maria/unittest/ma_test_loghandler_max_lsn-t.c''
      d0398f72
  11. 13 Aug, 2007 3 commits
    • unknown's avatar
      First LSN calls added for transaction log. · 4cf6756e
      unknown authored
      storage/maria/ma_checkpoint.c:
        Definitions of LSN should be collected in
        the one file (ma_loghandler_lsn.h)
      storage/maria/ma_loghandler.c:
        New calls to get first theoretical and first real LSN.
      storage/maria/ma_loghandler.h:
        New calls to get first theoretical and first real LSN.
      storage/maria/ma_loghandler_lsn.h:
        Defined yet another impossible LSN to indicate error.
      storage/maria/ma_recovery.c:
        The first LSN call changed.
      storage/maria/maria_read_log.c:
        The first LSN call changed.
      storage/maria/unittest/Makefile.am:
        New unittest added.
      storage/maria/unittest/ma_test_loghandler_first_lsn-t.c:
        New BitKeeper file ``storage/maria/unittest/ma_test_loghandler_first_lsn-t.c''
      4cf6756e
    • unknown's avatar
      Merge desktop.sanja.is.com.ua:/home/bell/mysql/bk/work-maria-noflush · fddf5050
      unknown authored
      into  desktop.sanja.is.com.ua:/home/bell/mysql/bk/work-maria
      
      
      storage/maria/ma_pagecache.c:
        Auto merged
      storage/maria/unittest/ma_test_loghandler-t.c:
        Auto merged
      storage/maria/ma_loghandler.c:
        merge
      storage/maria/ma_pagecache.h:
        merge
      fddf5050
    • unknown's avatar
      Ability to read unflushed data added (only problem with CRC left and have to be fixed). · 7c32eac4
      unknown authored
      Already written pages injection to the cache fixed.
      
      
      storage/maria/ma_loghandler.c:
        Ability to read unflushed data added.
      storage/maria/ma_page.c:
        Parameters added
      storage/maria/ma_pagecache.c:
        Already written pages injection to the cache fixed.
        Validator for case of page content injection added.
      storage/maria/ma_pagecache.h:
        Validator for case of page content injection added.
      storage/maria/unittest/Makefile.am:
        Test of reading unflushed data
      storage/maria/unittest/ma_test_loghandler-t.c:
        Define fixed.
        Restart of the log removed.
      storage/maria/unittest/ma_test_loghandler_noflush-t.c:
        New BitKeeper file ``storage/maria/unittest/ma_test_loghandler_noflush-t.c''
      7c32eac4
  12. 10 Aug, 2007 1 commit
  13. 09 Aug, 2007 2 commits
    • unknown's avatar
      * tests which use MERGE or INSERT DELAYED should run only · 0d301ee8
      unknown authored
      with engines which support that
      * temporarily adding option --global-subst to mysqltest so that
      the full testsuite can be run using Maria tables without failing
      on trivial differences (like diff in the engine clause of
      SHOW CREATE TABLE)
      * using recognizable tags for todos of the Maria team
      
      
      client/mysqltest.c:
        temporarily adding option --global-subst: its argument is X,Y.
        It replaces all occurrences of X by Y into mysqltest's result
        before the comparison with the expected result is done.
        This serves for when a test is run with --default-storage-engine=X
        where X is not MyISAM: tests using SHOW CREATE TABLE will always fail
        because SHOW CREATE TABLE prints X instead of MyISAM. With
        --global-subst=X,MyISAM , such trivial differences are eliminated and
        test may be reported as passing.
        For example, --global-subst=MARIA,MyISAM
        This is not good enough for merging into main trees! just for running
        many tests and finding bugs now!
      mysql-test/mysql-test-run.pl:
        new option --mysqltest to pass options to mysqltest (like we have
        --mysqld). Used for example like this:
        ./mtr --mysqltest=--global-subst=MARIA,MyISAM
      mysql-test/r/merge.result:
        update
      mysql-test/t/delayed.test:
        run test only with engines which support INSERT DELAYED
      mysql-test/t/merge.test:
        run test only with MyISAM tables, as they are required by MERGE
      sql/sql_delete.cc:
        recognizable tag
      sql/table.cc:
        recognizable tag
      storage/maria/ha_maria.cc:
        recognizable tag
      storage/maria/ma_check.c:
        recognizable tag
      storage/maria/ma_create.c:
        recognizable tag
      0d301ee8
    • unknown's avatar
      * "transactionality" needs to be preserved by TRUNCATE TABLE: · 95e2558f
      unknown authored
      a table with TRANSACTIONAL=x needs to still have it after TRUNCATE.
      No testcase, but without this fix, the frm and the Maria table
      got "out of sync" in this case:
      create table t1 (a int) row_format=page transactional=0;
      truncate table t1;
      After TRUNCATE, the Maria table (not the frm) was transactional
      (thus logging records, which is wrong).
      * fix for non-closed file at end of "maria_chk -r"
      
      
      sql/table.cc:
        "transactionality" needs to be preserved when truncating.
        It's behind a if() to not cancel the hack added to mysql_truncate()
        today for temporary Maria tables.
      storage/maria/ha_maria.cc:
        question for Monty (he also has a big mail from me on the same subject)
      storage/maria/ma_check.c:
        question for Monty (likely bugs)
      storage/maria/ma_create.c:
        debugging info
      storage/maria/ma_open.c:
        fix for datafile left open at end of "maria_chk -r":
        ma_open_datafile() happens after _ma_bitmap_init(), it sets dfile.file
        so needs to set share->bitmap.file.file too (they are copies of
        each other). Otherwise it breaks how closing of files works in
        BLOCK_RECORD (which is that info.dfile.file is not closed
        but share->bitmap.file.file is closed): not setting share->bitmap.file.file
        can lead to forgetting to close a file or closing a wrong file.
      95e2558f
  14. 08 Aug, 2007 3 commits
    • unknown's avatar
      as I disabled creation of the first empty bitmap at creation time, · 83ea6e4f
      unknown authored
      I disable it too for repair time.
      
      
      83ea6e4f
    • unknown's avatar
      Fix for · ddac4525
      unknown authored
      ./mtr --mysqld=--default-storage-engine=maria --mem ps:
      I got "can't sync on file UNOPENED" among the messages of REPAIR TABLE;
      due to a missing setting of bitmap.file.file to -1.
      Maria had two names "Maria" and "MARIA", using one now: "MARIA".
      
      
      storage/maria/ha_maria.cc:
        plug.in uses "MARIA". Some code apparently picks the name from
        plug.in (SHOW CREATE TABLE, run ps.test on Maria tables), other from
        mysql_declare_plugin (SHOW CREATE TABLE on partitioned tables, run
        partition.test with Maria tables), better make names identical.
      storage/maria/ma_check.c:
        running ps.test on Maria tables I got "can't sync on file UNOPENED"
        among the messages of REPAIR TABLE. That was due to maria_repair()
        closing the data file, setting info->dfile.file to -1 to prevent
        a wrong double close, but forgetting to also set
        info->s->bitmap.file.file to -1; it left it unchanged and so,
        when close_thread_tables() closed the old version of the repaired
        table, _ma_once_end_block_record() tried to fsync the closed
        descriptor, resulting in a message.
        Basically, when setting info->dfile.file to something it's always
        safe and recommended to set bitmap.file.file to the same value
        as it's just a copy of the same descriptor see _ma_bitmap_init().
        Using change_data_file_descriptor() for that.
        Changing that function to use MY_WME as it looks safe.
      storage/maria/ma_close.c:
        no need to make the index file durable if table is not transactional
      ddac4525
    • unknown's avatar
      Tests which use INSERT DELAYED or MERGE tables should explicitely · c7f9cf12
      unknown authored
      specify that they want a MyISAM table, otherwise they fail
      when run with --default-storage-engine=maria.
      
      
      mysql-test/extra/rpl_tests/rpl_flsh_tbls.test:
        uses INSERT DELAYED so has to specify MyISAM
      mysql-test/extra/rpl_tests/rpl_insert_delayed.test:
        uses INSERT DELAYED so has to specify MyISAM
      mysql-test/r/query_cache.result:
        result update
      mysql-test/r/rpl_insert.result:
        result update
      mysql-test/r/rpl_stm_flsh_tbls.result:
        result update
      mysql-test/r/rpl_stm_insert_delayed.result:
        result update
      mysql-test/r/rpl_switch_stm_row_mixed.result:
        result update
      mysql-test/r/subselect.result:
        result update
      mysql-test/t/query_cache.test:
        uses MERGE so has to specify MyISAM
      mysql-test/t/query_cache_merge.test:
        uses MERGE so has to specify MyISAM
      mysql-test/t/rpl_insert.test:
        uses INSERT DELAYED so has to specify MyISAM
      mysql-test/t/rpl_stm_flsh_tbls.test:
        specifying the engine lengthens the query in binlog so shifts
        positions
      mysql-test/t/rpl_switch_stm_row_mixed.test:
        uses INSERT DELAYED so has to specify MyISAM
      mysql-test/t/subselect.test:
        uses INSERT DELAYED so has to specify MyISAM
      c7f9cf12