An error occurred fetching the project authors.
  1. 20 Mar, 2018 1 commit
    • Marko Mäkelä's avatar
      MDEV-12396 IMPORT TABLESPACE cleanup · 6247c64c
      Marko Mäkelä authored
      Reduce unnecessary inter-module calls for IMPORT TABLESPACE.
      Move some IMPORT-related code from fil0fil.cc to row0import.cc.
      
      PageCallback: Remove. Make AbstractCallback the base class.
      
      PageConverter: Define some member functions inline.
      6247c64c
  2. 08 Feb, 2018 1 commit
  3. 06 Nov, 2017 1 commit
    • Marko Mäkelä's avatar
      MDEV-13328 ALTER TABLE…DISCARD TABLESPACE takes a lot of time · 51b4366b
      Marko Mäkelä authored
      With a big buffer pool that contains many data pages,
      DISCARD TABLESPACE took a long time, because it would scan the
      entire buffer pool to remove any pages that belong to the tablespace.
      With a large buffer pool, this would take a lot of time, especially
      when the table-to-discard is empty.
      
      The minimum amount of work that DISCARD TABLESPACE must do is to
      remove the pages of the to-be-discarded table from the
      buf_pool->flush_list because any writes to the data file must be
      prevented before the file is deleted.
      
      If DISCARD TABLESPACE does not evict the pages from the buffer pool,
      then IMPORT TABLESPACE must do it, because we must prevent pre-DISCARD,
      not-yet-evicted pages from being mistaken for pages of the imported
      tablespace.
      
      It would not be a useful fix to simply move the buffer pool scan to
      the IMPORT TABLESPACE step. What we can do is to actively evict those
      pages that could be mistaken for imported pages. In this way, when
      importing a small table into a big buffer pool, the import should
      still run relatively fast.
      
      Import is bypassing the buffer pool when reading pages for the
      adjustment phase. In the adjustment phase, if a page exists in
      the buffer pool, we could replace it with the page from the imported
      file. Unfortunately I did not get this to work properly, so instead
      we will simply evict any matching page from the buffer pool.
      
      buf_page_get_gen(): Implement BUF_EVICT_IF_IN_POOL, a new mode
      where the requested page will be evicted if it is found. There
      must be no unwritten changes for the page.
      
      buf_remove_t: Remove. Instead, use trx!=NULL to signify that a write
      to file is desired, and use a separate parameter bool drop_ahi.
      
      buf_LRU_flush_or_remove_pages(), fil_delete_tablespace():
      Replace buf_remove_t.
      
      buf_LRU_remove_pages(), buf_LRU_remove_all_pages(): Remove.
      
      PageConverter::m_mtr: A dummy mini-transaction buffer
      
      PageConverter::PageConverter(): Complete the member initialization list.
      
      PageConverter::operator()(): Evict any 'shadow' pages from the
      buffer pool so that pre-existing (garbage) pages cannot be mistaken
      for pages that exist in the being-imported file.
      
      row_discard_tablespace(): Remove a bogus comment that seems to
      refer to IMPORT TABLESPACE, not DISCARD TABLESPACE.
      51b4366b
  4. 24 Sep, 2017 1 commit
    • Marko Mäkelä's avatar
      MDEV-13899 IMPORT TABLESPACE may corrupt ROW_FORMAT=REDUNDANT tables · 78b63425
      Marko Mäkelä authored
      The ALTER TABLE…IMPORT TABLESPACE adjustment code that was introduced by
      WL#5522 in MySQL 5.6 is incorrectly invoking rec_get_status() on a
      ROW_FORMAT=REDUNDANT record to determine if a record is a leaf page record.
      The function rec_get_status(rec) is only to be called on
      ROW_FORMAT=COMPACT, DYNAMIC or COMPRESSED records.
      78b63425
  5. 05 Jul, 2017 1 commit
  6. 15 Jun, 2017 1 commit
    • Marko Mäkelä's avatar
      Remove some fields from dict_table_t · 58f87a41
      Marko Mäkelä authored
      dict_table_t::thd: Remove. This was only used by btr_root_block_get()
      for reporting decryption failures, and it was only assigned by
      ha_innobase::open(), and never cleared. This could mean that if a
      connection is closed, the pointer would become stale, and the server
      could crash while trying to report the error. It could also mean
      that an error is being reported to the wrong client. It is better
      to use current_thd in this case, even though it could mean that if
      the code is invoked from an InnoDB background operation, there would
      be no connection to which to send the error message.
      
      Remove dict_table_t::crypt_data and dict_table_t::page_0_read.
      These fields were never read.
      
      fil_open_single_table_tablespace(): Remove the parameter "table".
      58f87a41
  7. 17 May, 2017 2 commits
  8. 26 Apr, 2017 1 commit
    • Jan Lindström's avatar
      MDEV-12253: Buffer pool blocks are accessed after they have been freed · 765a4360
      Jan Lindström authored
      Problem was that bpage was referenced after it was already freed
      from LRU. Fixed by adding a new variable encrypted that is
      passed down to buf_page_check_corrupt() and used in
      buf_page_get_gen() to stop processing page read.
      
      This patch should also address following test failures and
      bugs:
      
      MDEV-12419: IMPORT should not look up tablespace in
      PageConverter::validate(). This is now removed.
      
      MDEV-10099: encryption.innodb_onlinealter_encryption fails
      sporadically in buildbot
      
      MDEV-11420: encryption.innodb_encryption-page-compression
      failed in buildbot
      
      MDEV-11222: encryption.encrypt_and_grep failed in buildbot on P8
      
      Removed dict_table_t::is_encrypted and dict_table_t::ibd_file_missing
      and replaced these with dict_table_t::file_unreadable. Table
      ibd file is missing if fil_get_space(space_id) returns NULL
      and encrypted if not. Removed dict_table_t::is_corrupted field.
      
      Ported FilSpace class from 10.2 and using that on buf_page_check_corrupt(),
      buf_page_decrypt_after_read(), buf_page_encrypt_before_write(),
      buf_dblwr_process(), buf_read_page(), dict_stats_save_defrag_stats().
      
      Added test cases when enrypted page could be read while doing
      redo log crash recovery. Also added test case for row compressed
      blobs.
      
      btr_cur_open_at_index_side_func(),
      btr_cur_open_at_rnd_pos_func(): Avoid referencing block that is
      NULL.
      
      buf_page_get_zip(): Issue error if page read fails.
      
      buf_page_get_gen(): Use dberr_t for error detection and
      do not reference bpage after we hare freed it.
      
      buf_mark_space_corrupt(): remove bpage from LRU also when
      it is encrypted.
      
      buf_page_check_corrupt(): @return DB_SUCCESS if page has
      been read and is not corrupted,
      DB_PAGE_CORRUPTED if page based on checksum check is corrupted,
      DB_DECRYPTION_FAILED if page post encryption checksum matches but
      after decryption normal page checksum does not match. In read
      case only DB_SUCCESS is possible.
      
      buf_page_io_complete(): use dberr_t for error handling.
      
      buf_flush_write_block_low(),
      buf_read_ahead_random(),
      buf_read_page_async(),
      buf_read_ahead_linear(),
      buf_read_ibuf_merge_pages(),
      buf_read_recv_pages(),
      fil_aio_wait():
              Issue error if page read fails.
      
      btr_pcur_move_to_next_page(): Do not reference page if it is
      NULL.
      
      Introduced dict_table_t::is_readable() and dict_index_t::is_readable()
      that will return true if tablespace exists and pages read from
      tablespace are not corrupted or page decryption failed.
      Removed buf_page_t::key_version. After page decryption the
      key version is not removed from page frame. For unencrypted
      pages, old key_version is removed at buf_page_encrypt_before_write()
      
      dict_stats_update_transient_for_index(),
      dict_stats_update_transient()
              Do not continue if table decryption failed or table
              is corrupted.
      
      dict0stats.cc: Introduced a dict_stats_report_error function
      to avoid code duplication.
      
      fil_parse_write_crypt_data():
              Check that key read from redo log entry is found from
              encryption plugin and if it is not, refuse to start.
      
      PageConverter::validate(): Removed access to fil_space_t as
      tablespace is not available during import.
      
      Fixed error code on innodb.innodb test.
      
      Merged test cased innodb-bad-key-change5 and innodb-bad-key-shutdown
      to innodb-bad-key-change2.  Removed innodb-bad-key-change5 test.
      Decreased unnecessary complexity on some long lasting tests.
      
      Removed fil_inc_pending_ops(), fil_decr_pending_ops(),
      fil_get_first_space(), fil_get_next_space(),
      fil_get_first_space_safe(), fil_get_next_space_safe()
      functions.
      
      fil_space_verify_crypt_checksum(): Fixed bug found using ASAN
      where FIL_PAGE_END_LSN_OLD_CHECKSUM field was incorrectly
      accessed from row compressed tables. Fixed out of page frame
      bug for row compressed tables in
      fil_space_verify_crypt_checksum() found using ASAN. Incorrect
      function was called for compressed table.
      
      Added new tests for discard, rename table and drop (we should allow them
      even when page decryption fails). Alter table rename is not allowed.
      Added test for restart with innodb-force-recovery=1 when page read on
      redo-recovery cant be decrypted. Added test for corrupted table where
      both page data and FIL_PAGE_FILE_FLUSH_LSN_OR_KEY_VERSION is corrupted.
      
      Adjusted the test case innodb_bug14147491 so that it does not anymore
      expect crash. Instead table is just mostly not usable.
      
      fil0fil.h: fil_space_acquire_low is not visible function
      and fil_space_acquire and fil_space_acquire_silent are
      inline functions. FilSpace class uses fil_space_acquire_low
      directly.
      
      recv_apply_hashed_log_recs() does not return anything.
      765a4360
  9. 21 Apr, 2017 1 commit
    • Marko Mäkelä's avatar
      MDEV-12488 Remove type mismatch in InnoDB printf-like calls · 87b6df31
      Marko Mäkelä authored
      This is a reduced version of an originally much larger patch.
      We will keep the definition of the ulint, lint data types unchanged,
      and we will not be replacing fprintf() calls with ib_logf().
      
      On Windows, use the standard format strings instead of nonstandard
      extensions.
      
      This patch fixes some errors in format strings.
      Most notably, an IMPORT TABLESPACE error message in InnoDB was
      displaying the number of columns instead of the mismatching flags.
      87b6df31
  10. 09 Mar, 2017 1 commit
  11. 06 Feb, 2017 1 commit
    • Jan Lindström's avatar
      MDEV-11759: Encryption code in MariaDB 10.1/10.2 causes · ddf2fac7
      Jan Lindström authored
      compatibility problems
      
      Pages that are encrypted contain post encryption checksum on
      different location that normal checksum fields. Therefore,
      we should before decryption check this checksum to avoid
      unencrypting corrupted pages. After decryption we can use
      traditional checksum check to detect if page is corrupted
      or unencryption was done using incorrect key.
      
      Pages that are page compressed do not contain any checksum,
      here we need to fist unencrypt, decompress and finally
      use tradional checksum check to detect page corruption
      or that we used incorrect key in unencryption.
      
      buf0buf.cc: buf_page_is_corrupted() mofified so that
      compressed pages are skipped.
      
      buf0buf.h, buf_block_init(), buf_page_init_low():
      removed unnecessary page_encrypted, page_compressed,
      stored_checksum, valculated_checksum fields from
      buf_page_t
      
      buf_page_get_gen(): use new buf_page_check_corrupt() function
      to detect corrupted pages.
      
      buf_page_check_corrupt(): If page was not yet decrypted
      check if post encryption checksum still matches.
      If page is not anymore encrypted, use buf_page_is_corrupted()
      traditional checksum method.
      
      If page is detected as corrupted and it is not encrypted
      we print corruption message to error log.
      If page is still encrypted or it was encrypted and now
      corrupted, we will print message that page is
      encrypted to error log.
      
      buf_page_io_complete(): use new buf_page_check_corrupt()
      function to detect corrupted pages.
      
      buf_page_decrypt_after_read(): Verify post encryption
      checksum before tring to decrypt.
      
      fil0crypt.cc: fil_encrypt_buf() verify post encryption
      checksum and ind fil_space_decrypt() return true
      if we really decrypted the page.
      
      fil_space_verify_crypt_checksum(): rewrite to use
      the method used when calculating post encryption
      checksum. We also check if post encryption checksum
      matches that traditional checksum check does not
      match.
      
      fil0fil.ic: Add missed page type encrypted and page
      compressed to fil_get_page_type_name()
      
      Note that this change does not yet fix innochecksum tool,
      that will be done in separate MDEV.
      
      Fix test failures caused by buf page corruption injection.
      ddf2fac7
  12. 15 Jan, 2017 1 commit
    • Marko Mäkelä's avatar
      MDEV-11623 MariaDB 10.1 fails to start datadir created with · ab1e6fef
      Marko Mäkelä authored
      MariaDB 10.0/MySQL 5.6 using innodb-page-size!=16K
      
      The storage format of FSP_SPACE_FLAGS was accidentally broken
      already in MariaDB 10.1.0. This fix is bringing the format in
      line with other MySQL and MariaDB release series.
      
      Please refer to the comments that were added to fsp0fsp.h
      for details.
      
      This is an INCOMPATIBLE CHANGE that affects users of
      page_compression and non-default innodb_page_size. Upgrading
      to this release will correct the flags in the data files.
      If you want to downgrade to earlier MariaDB 10.1.x, please refer
      to the test innodb.101_compatibility how to reset the
      FSP_SPACE_FLAGS in the files.
      
      NOTE: MariaDB 10.1.0 to 10.1.20 can misinterpret
      uncompressed data files with innodb_page_size=4k or 64k as
      compressed innodb_page_size=16k files, and then probably fail
      when trying to access the pages. See the comments in the
      function fsp_flags_convert_from_101() for detailed analysis.
      
      Move PAGE_COMPRESSION to FSP_SPACE_FLAGS bit position 16.
      In this way, compressed innodb_page_size=16k tablespaces will not
      be mistaken for uncompressed ones by MariaDB 10.1.0 to 10.1.20.
      
      Derive PAGE_COMPRESSION_LEVEL, ATOMIC_WRITES and DATA_DIR from the
      dict_table_t::flags when the table is available, in
      fil_space_for_table_exists_in_mem() or fil_open_single_table_tablespace().
      During crash recovery, fil_load_single_table_tablespace() will use
      innodb_compression_level for the PAGE_COMPRESSION_LEVEL.
      
      FSP_FLAGS_MEM_MASK: A bitmap of the memory-only fil_space_t::flags
      that are not to be written to FSP_SPACE_FLAGS. Currently, these will
      include PAGE_COMPRESSION_LEVEL, ATOMIC_WRITES and DATA_DIR.
      
      Introduce the macro FSP_FLAGS_PAGE_SSIZE(). We only support
      one innodb_page_size for the whole instance.
      
      When creating a dummy tablespace for the redo log, use
      fil_space_t::flags=0. The flags are never written to the redo log files.
      
      Remove many FSP_FLAGS_SET_ macros.
      
      dict_tf_verify_flags(): Remove. This is basically only duplicating
      the logic of dict_tf_to_fsp_flags(), used in a debug assertion.
      
      fil_space_t::mark: Remove. This flag was not used for anything.
      
      fil_space_for_table_exists_in_mem(): Remove the unnecessary parameter
      mark_space, and add a parameter for table flags. Check that
      fil_space_t::flags match the table flags, and adjust the (memory-only)
      flags based on the table flags.
      
      fil_node_open_file(): Remove some redundant or unreachable conditions,
      do not use stderr for output, and avoid unnecessary server aborts.
      
      fil_user_tablespace_restore_page(): Convert the flags, so that the
      correct page_size will be used when restoring a page from the
      doublewrite buffer.
      
      fil_space_get_page_compressed(), fsp_flags_is_page_compressed(): Remove.
      It suffices to have fil_space_is_page_compressed().
      
      FSP_FLAGS_WIDTH_DATA_DIR, FSP_FLAGS_WIDTH_PAGE_COMPRESSION_LEVEL,
      FSP_FLAGS_WIDTH_ATOMIC_WRITES: Remove, because these flags do not
      exist in the FSP_SPACE_FLAGS but only in memory.
      
      fsp_flags_try_adjust(): New function, to adjust the FSP_SPACE_FLAGS
      in page 0. Called by fil_open_single_table_tablespace(),
      fil_space_for_table_exists_in_mem(), innobase_start_or_create_for_mysql()
      except if --innodb-read-only is active.
      
      fsp_flags_is_valid(ulint): Reimplement from the scratch, with
      accurate comments. Do not display any details of detected
      inconsistencies, because the output could be confusing when
      dealing with MariaDB 10.1.x data files.
      
      fsp_flags_convert_from_101(ulint): Convert flags from buggy
      MariaDB 10.1.x format, or return ULINT_UNDEFINED if the flags
      cannot be in MariaDB 10.1.x format.
      
      fsp_flags_match(): Check the flags when probing files.
      Implemented based on fsp_flags_is_valid()
      and fsp_flags_convert_from_101().
      
      dict_check_tablespaces_and_store_max_id(): Do not access the
      page after committing the mini-transaction.
      
      IMPORT TABLESPACE fixes:
      
      AbstractCallback::init(): Convert the flags.
      
      FetchIndexRootPages::operator(): Check that the tablespace flags match the
      table flags. Do not attempt to convert tablespace flags to table flags,
      because the conversion would necessarily be lossy.
      
      PageConverter::update_header(): Write back the correct flags.
      This takes care of the flags in IMPORT TABLESPACE.
      ab1e6fef
  13. 29 Oct, 2016 1 commit
  14. 21 Jun, 2016 1 commit
  15. 13 Dec, 2015 1 commit
  16. 14 Sep, 2015 1 commit
  17. 04 Sep, 2015 1 commit
    • Jan Lindström's avatar
      MDEV-8588: Assertion failure in file ha_innodb.cc line 21140 if at least one... · e1978234
      Jan Lindström authored
      MDEV-8588: Assertion failure in file ha_innodb.cc line 21140 if at least one encrypted table exists and encryption service is not available
      
      Analysis: Problem was that in fil_read_first_page we do find that
      table has encryption information and that encryption service
      or used key_id is not available. But, then we just printed
      fatal error message that causes above assertion.
      
      Fix: When we open single table tablespace if it has encryption
      information (crypt_data) store this crypt data to the table
      structure. When we open a table and we find out that tablespace
      is not available, check has table a encryption information
      and from there is encryption service or used key_id is not available.
      If it is, add additional warning for SQL-layer.
      e1978234
  18. 03 Aug, 2015 1 commit
  19. 04 May, 2015 2 commits
  20. 26 Feb, 2014 1 commit
  21. 16 Dec, 2013 1 commit