An error occurred fetching the project authors.
  1. 13 Nov, 2020 1 commit
    • Marko Mäkelä's avatar
      MDEV-24188 Hang in buf_page_create() after reusing a previously freed page · bb328a2a
      Marko Mäkelä authored
      The fix of MDEV-23456 (commit b1009ae5)
      introduced a livelock between page flushing and a thread that is
      executing buf_page_create().
      
      buf_page_create(): If the current mini-transaction is holding
      an exclusive latch on the page, do not attempt to acquire another
      one, and do not care about any I/O fix.
      
      mtr_t::have_x_latch(): Replaces mtr_t::get_fix_count().
      
      dyn_buf_t::for_each_block(const Functor&) const: A new variant.
      
      rw_lock_own(): Add a const qualifier.
      
      Reviewed by: Thirunarayanan Balathandayuthapani
      bb328a2a
  2. 11 Nov, 2020 1 commit
    • Marko Mäkelä's avatar
      MDEV-24182 ibuf_merge_or_delete_for_page() contains dead code · bd528b0c
      Marko Mäkelä authored
      The function ibuf_merge_or_delete_for_page() was always being
      invoked with update_ibuf_bitmap=true ever since
      commit cd623508
      fixed up something after MDEV-9566.
      
      Furthermore, the parameter page_size is never being passed as a
      null pointer, and therefore it should better be a reference to
      a constant object.
      bd528b0c
  3. 27 Oct, 2020 1 commit
    • Thirunarayanan Balathandayuthapani's avatar
      MDEV-23693 Failing assertion: my_atomic_load32_explicit(&lock->lock_word,... · bc540b87
      Thirunarayanan Balathandayuthapani authored
      MDEV-23693 Failing assertion: my_atomic_load32_explicit(&lock->lock_word, MY_MEMORY_ORDER_RELAXED) == X_LOCK_DECR
      
      InnoDB frees the block lock during buffer pool shrinking when other
      thread is yet to release the block lock.  While shrinking the
      buffer pool, InnoDB allows the page to be freed unless it is buffer
      fixed. In some cases, InnoDB releases the latch after unfixing the
      block.
      
      Fix:
      ====
      - InnoDB should unfix the block after releases the latch.
      
      - Add more assertion to check buffer fix while accessing the page.
      
      - Introduced block_hint structure to store buf_block_t pointer
      and allow accessing the buf_block_t pointer only by passing a
      functor. It returns original buf_block_t* pointer if it is valid
      or nullptr if the pointer become stale.
      
      - Replace buf_block_is_uncompressed() with
      buf_pool_t::is_block_pointer()
      
      This change is motivated by a change in mysql-5.7.32:
      mysql/mysql-server@46e60de444a8fbd876cc6778a7e64a1d3426a48d
      Bug #31036301 ASSERTION FAILURE: SYNC0RW.IC:429:LOCK->LOCK_WORD
      bc540b87
  4. 09 Sep, 2020 1 commit
    • Thirunarayanan Balathandayuthapani's avatar
      MDEV-23456 fil_space_crypt_t::write_page0() is accessing an uninitialized page · b1009ae5
      Thirunarayanan Balathandayuthapani authored
      buf_page_create() is invoked when page is initialized. So that
      previous contents of the page ignored. In few cases, it calls
      buf_page_get_gen() is called to fetch the page from buffer pool.
      It should take x-latch on the page. If other thread uses the block
      or block io state is different from BUF_IO_NONE then release the
      mutex and check the state and buffer fix count again. For compressed
      page, use the existing free block from LRU list to create new page.
      Retry to fetch the compressed page if it is in flush list
      
      fseg_create(), fseg_create_general(): Introduce block as a parameter
      where segment header is placed. It is used to avoid repetitive
      x-latch on the same page
      
      Change the assert to check whether the page has SX latch and
      X latch in all callee function of buf_page_create()
      
      mtr_t::get_fix_count(): Get the buffer fix count of the given
      block added by the mtr
      
      FindBlock is added to find the buffer fix count of the given
      block acquired by the mini-transaction
      b1009ae5
  5. 20 Aug, 2020 2 commits
  6. 27 Jul, 2020 1 commit
    • Thirunarayanan Balathandayuthapani's avatar
      MDEV-23233: Race condition for btr_search_drop_page_hash_index() in buf_page_create() · a1f899a8
      Thirunarayanan Balathandayuthapani authored
      commit ad6171b9 (MDEV-22456)
      introduced code to buf_page_create() that would lazily drop
      adaptive hash index entries for an index that has been
      evicted from the data dictionary cache.
      
      Unfortunately, that call was missing adequate protection.
      While the btr_search_drop_page_hash_index(block) was executing,
      the block could be reused for something else.
      
      buf_page_create(): If btr_search_drop_page_hash_index() must be
      invoked, pin the block before releasing the buf_pool->page_hash lock,
      so that the block cannot be grabbed by other threads.
      a1f899a8
  7. 16 Jul, 2020 1 commit
    • Marko Mäkelä's avatar
      MDEV-21347 innodb_log_optimize_ddl=OFF is not crash safe · 147d4b1e
      Marko Mäkelä authored
      In commit 0f90728b (MDEV-16809)
      we introduced the configuration option innodb_log_optimize_ddl
      for controlling whether native index creation or table-rebuild
      in InnoDB should avoid writing full redo log.
      
      Fungo Wang reported that this option is causing occasional failures.
      The reason is that pages may be written to data files in an
      inconsistent state. Applying log records to such inconsistent pages
      may fail.
      
      The solution is to always invoke PageBulk::finish() before page latches
      may be released, to ensure that the page contents is in a consistent
      state.
      
      Something similar was implemented in MySQL 8.0.13:
      mysql/mysql-server@d1254b947354e0f5b7223b09c521bd85f22e1e31
      
      buf_block_t::skip_flush_check: Remove. Suppressing consistency checks
      is a bad idea.
      
      PageBulk::needs_finish(): New predicate: Determine whether
      PageBulk::finish() must fix up the page.
      
      PageBulk::init(): Clear PAGE_DIRECTION to ensure that needs_finish()
      will hold. We change the field from PAGE_NO_DIRECTION to 0
      and back without writing redo log. This trick avoids the need
      to introduce any new data member to PageBulk.
      
      PageBulk::insert(): Replace some high-level accessors to bypass
      debug assertions related to PAGE_HEAP_TOP that we will be violating
      until finish() has been executed.
      
      PageBulk::finish(): Tolerate m_rec_no==0. We must invoke this also
      on an empty page, to ensure that PAGE_HEAP_TOP is initialized.
      
      PageBulk::commit(): Always invoke finish().
      
      PageBulk::release(), BtrBulk::pageSplit(), BtrBulk::storeExt(),
      BtrBulk::finish(): Invoke PageBulk::finish().
      147d4b1e
  8. 05 Jul, 2020 1 commit
    • Marko Mäkelä's avatar
      One more ASAN/MSAN cleanup · f3f23b5c
      Marko Mäkelä authored
      commit 48493132 included a
      workaround for a 10.5 merge issue that should now be properly
      addressed in commit ab406990.
      
      buf_chunk_init(): Remove an unnecessary MEM_MAKE_ADDRESSABLE().
      We might invoke MEM_UNDEFINED() here, but actually the allocated
      memory ought to be guaranteed to be zero-initialized.
      f3f23b5c
  9. 04 Jul, 2020 1 commit
    • Marko Mäkelä's avatar
      Fixup the parent commit for MSAN and Valgrind · 453dc4b3
      Marko Mäkelä authored
      commit 48493132 was a necessary
      fix for the buffer pool resizing tests in 10.5 in
      AddressSanitizer. However, that change would break the tests
      innodb.innodb_buffer_pool_resize and
      innodb.innodb_buffer_pool_resize_with_chunks
      when run in MemorySanitizer, or presumably in Valgrind as well.
      (Those tests run "forever" in Valgrind.)
      
      buf_pool_resize(): Cancel the effect of MEM_NOACCESS() in Valgrind
      and ASAN. In MSAN, MEM_NOACCESS() is a no-op, and hence we must do
      nothing special here.
      
      MEM_MAKE_ADDRESSABLE() would declare the memory contents undefined.
      In this particular case, we must actually declare the contents
      defined for Valgrind.
      453dc4b3
  10. 03 Jul, 2020 1 commit
  11. 01 Jul, 2020 1 commit
    • Marko Mäkelä's avatar
      MDEV-20377: Make WITH_MSAN more usable · c36834c8
      Marko Mäkelä authored
      MemorySanitizer (clang -fsanitize=memory) requires that all code
      be compiled with instrumentation enabled. The only exception is the
      C runtime library. Failure to use instrumented libraries will cause
      bogus messages about memory being uninitialized.
      
      In WITH_MSAN builds, we must avoid calling getservbyname(),
      because even though it is a standard library function, it is
      not instrumented, not even in clang 10.
      
      Note: Before MariaDB Server 10.5, ./mtr will typically fail
      due to the old PCRE library, which was updated in MDEV-14024.
      
      The following cmake options were tested on 10.5
      in commit 94d0bb4d:
      
      cmake \
      -DCMAKE_C_FLAGS='-march=native -O2' \
      -DCMAKE_CXX_FLAGS='-stdlib=libc++ -march=native -O2' \
      -DWITH_EMBEDDED_SERVER=OFF -DWITH_UNIT_TESTS=OFF -DCMAKE_BUILD_TYPE=Debug \
      -DWITH_INNODB_{BZIP2,LZ4,LZMA,LZO,SNAPPY}=OFF \
      -DPLUGIN_{ARCHIVE,TOKUDB,MROONGA,OQGRAPH,ROCKSDB,CONNECT,SPIDER}=NO \
      -DWITH_SAFEMALLOC=OFF \
      -DWITH_{ZLIB,SSL,PCRE}=bundled \
      -DHAVE_LIBAIO_H=0 \
      -DWITH_MSAN=ON
      
      MEM_MAKE_DEFINED(): An alias for VALGRIND_MAKE_MEM_DEFINED()
      and __msan_unpoison().
      
      MEM_GET_VBITS(), MEM_SET_VBITS(): Aliases for
      VALGRIND_GET_VBITS(), VALGRIND_SET_VBITS(), __msan_copy_shadow().
      
      InnoDB: Replace the UNIV_MEM_ macros with corresponding MEM_ macros.
      
      ut_crc32_8_hw(), ut_crc32_64_low_hw(): Use the compiler built-in
      functions instead of inline assembler when building WITH_MSAN.
      This will require at least -msse4.2 when building for IA-32 or AMD64.
      The inline assembler would not be instrumented, and would thus cause
      bogus failures.
      c36834c8
  12. 05 Jun, 2020 1 commit
    • Marko Mäkelä's avatar
      MDEV-22790 Race between btr_page_mtr_lock() dropping AHI on the same block · 138c11cc
      Marko Mäkelä authored
      This race condition was introduced by
      commit ad6171b9 (MDEV-22456).
      
      In the observed case, two threads were executing
      btr_search_drop_page_hash_index() on the same block,
      to free a stale entry that was attached to a dropped index.
      Both threads were holding an S latch on the block.
      
      We must prevent the double-free of block->index by holding
      block->lock in exclusive mode.
      
      btr_search_guess_on_hash(): Do not invoke
      btr_search_drop_page_hash_index(block) to get rid of
      stale entries, because we are not necessarily holding
      an exclusive block->lock here.
      
      buf_defer_drop_ahi(): New function, to safely drop stale
      entries in buf_page_mtr_lock(). We will skip the call to
      btr_search_drop_page_hash_index(block) when only requesting
      bufferfixing (no page latch), because in that case, we should
      not be accessing the adaptive hash index, and we might get
      a deadlock if we acquired the page latch.
      138c11cc
  13. 04 Jun, 2020 1 commit
    • Marko Mäkelä's avatar
      MDEV-22721 Remove bloat caused by InnoDB logger class · eba2d10a
      Marko Mäkelä authored
      Introduce a new ATTRIBUTE_NOINLINE to
      ib::logger member functions, and add UNIV_UNLIKELY hints to callers.
      
      Also, remove some crash reporting output. If needed, the
      information will be available using debugging tools.
      
      Furthermore, remove some fts_enable_diag_print output that included
      indexed words in raw form. The code seemed to assume that words are
      NUL-terminated byte strings. It is not clear whether a NUL terminator
      is always guaranteed to be present. Also, UCS2 or UTF-16 strings would
      typically contain many NUL bytes.
      eba2d10a
  14. 03 Jun, 2020 1 commit
    • Thirunarayanan Balathandayuthapani's avatar
      MDEV-22646 Assertion `table2->cached' failed in dict_table_t::add_to_cache · ad2bf112
      Thirunarayanan Balathandayuthapani authored
      Problem:
      ========
        During buffer pool resizing, InnoDB recreates the dictionary hash
      tables. Dictionary hash table reuses the heap of AHI hash tables.
      It leads to memory corruption.
      
      Fix:
      ====
      - While disabling AHI, free the heap and AHI hash tables. Recreate the
      AHI hash tables and assign new heap when AHI is enabled.
      
      - btr_blob_free() access invalid page if page was reallocated during
      buffer poolresizing. So btr_blob_free() should get the page from
      buf_pool instead of using existing block.
      
      - btr_search_enabled and block->index should be checked after
      acquiring the btr_search_sys latch
      
      - Moved the buffer_pool_scan debug sync to earlier before accessing the
      btr_search_sys latches to avoid the hang of truncate_purge_debug
      test case
      
      - srv_printf_innodb_monitor() should acquire btr_search_sys latches
      before AHI hash tables.
      ad2bf112
  15. 01 Jun, 2020 1 commit
    • Thirunarayanan Balathandayuthapani's avatar
      MDEV-22650 Dirty compressed page checksum validation fails · 02f68552
      Thirunarayanan Balathandayuthapani authored
      Problem:
      =======
        While evicting the uncompressed page from buffer pool, InnoDB writes
      the checksum for the compressed page in buf_LRU_free_page().
      So while flushing the compressed page, checksum validation fails
      when innodb_checksum_algorithm variable changed to strict_none.
      
      Solution:
      ========
      - Calculate the checksum only during flushing of page. Removed the
      checksum write in buf_LRU_free_page().
      02f68552
  16. 15 May, 2020 1 commit
    • Marko Mäkelä's avatar
      MDEV-22456 Dropping the adaptive hash index may cause DDL to lock up InnoDB · ad6171b9
      Marko Mäkelä authored
      If the InnoDB buffer pool contains many pages for a table or index
      that is being dropped or rebuilt, and if many of such pages are
      pointed to by the adaptive hash index, dropping the adaptive hash index
      may consume a lot of time.
      
      The time-consuming operation of dropping the adaptive hash index entries
      is being executed while the InnoDB data dictionary cache dict_sys is
      exclusively locked.
      
      It is not actually necessary to drop all adaptive hash index entries
      at the time a table or index is being dropped or rebuilt. We can let
      the LRU replacement policy of the buffer pool take care of this gradually.
      For this to work, we must detach the dict_table_t and dict_index_t
      objects from the main dict_sys cache, and once the last
      adaptive hash index entry for the detached table is removed
      (when the garbage page is evicted from the buffer pool) we can free
      the dict_table_t and dict_index_t object.
      
      Related to this, in MDEV-16283, we made ALTER TABLE...DISCARD TABLESPACE
      skip both the buffer pool eviction and the drop of the adaptive hash index.
      We shifted the burden to ALTER TABLE...IMPORT TABLESPACE or DROP TABLE.
      We can remove the eviction from DROP TABLE. We must retain the eviction
      in the ALTER TABLE...IMPORT TABLESPACE code path, so that in case the
      discarded table is being re-imported with the same tablespace identifier,
      the fresh data from the imported tablespace will replace any stale pages
      in the buffer pool.
      
      rpl.rpl_failed_drop_tbl_binlog: Remove the test. DROP TABLE can
      no longer be interrupted inside InnoDB.
      
      fseg_free_page(), fseg_free_step(), fseg_free_step_not_header(),
      fseg_free_page_low(), fseg_free_extent(): Remove the parameter
      that specifies whether the adaptive hash index should be dropped.
      
      btr_search_lazy_free(): Lazily free an index when the last
      reference to it is dropped from the adaptive hash index.
      
      buf_pool_clear_hash_index(): Declare static, and move to the
      same compilation unit with the bulk of the adaptive hash index
      code.
      
      dict_index_t::clone(), dict_index_t::clone_if_needed():
      Clone an index that is being rebuilt while adaptive hash index
      entries exist. The original index will be inserted into
      dict_table_t::freed_indexes and dict_index_t::set_freed()
      will be called.
      
      dict_index_t::set_freed(), dict_index_t::freed(): Note that
      or check whether the index has been freed. We will use the
      impossible page number 1 to denote this condition.
      
      dict_index_t::n_ahi_pages(): Replaces btr_search_info_get_ref_count().
      
      dict_index_t::detach_columns(): Move the assignment n_fields=0
      to ha_innobase_inplace_ctx::clear_added_indexes().
      We must have access to the columns when freeing the
      adaptive hash index. Note: dict_table_t::v_cols[] will remain
      valid. If virtual columns are dropped or added, the table
      definition will be reloaded in ha_innobase::commit_inplace_alter_table().
      
      buf_page_mtr_lock(): Drop a stale adaptive hash index if needed.
      
      We will also reduce the number of btr_get_search_latch() calls
      and enclose some more code inside #ifdef BTR_CUR_HASH_ADAPT
      in order to benefit cmake -DWITH_INNODB_AHI=OFF.
      ad6171b9
  17. 14 May, 2020 1 commit
    • Marko Mäkelä's avatar
      MDEV-22070 MSAN use-of-uninitialized-value in encryption.innodb-redo-badkey · ee5152fc
      Marko Mäkelä authored
      On a checksum failure of a ROW_FORMAT=COMPRESSED page,
      buf_LRU_free_one_page() would invoke buf_LRU_block_remove_hashed()
      which will read the uncompressed page frame, although it would not
      be initialized. With bad enough luck, fil_page_get_type(page)
      could return an unrecognized value and cause the server to abort.
      
      buf_page_io_complete(): On the corruption of a ROW_FORMAT=COMPRESSED
      page, zerofill the uncompressed page frame.
      ee5152fc
  18. 05 May, 2020 1 commit
  19. 30 Apr, 2020 1 commit
    • Eugene Kosov's avatar
      split log_t::buf into two buffers · 7f9dc0d8
      Eugene Kosov authored
      Maybe this patch will help catch problems like buffer overflow.
      
      log_t::first_in_use: removed
      
      log_t::buf: this is where mtr_t are supposed to append data
      log_t::flush_buf: this is from server writes to a file
      
      Those two buffers are std::swap()ped when some thread is gonna write
      to a file
      7f9dc0d8
  20. 23 Mar, 2020 1 commit
    • Thirunarayanan Balathandayuthapani's avatar
      MDEV-21572 buf_page_get_gen() should apply buffered page initialized · 6697135c
      Thirunarayanan Balathandayuthapani authored
      		redo log during recovery
      
      - InnoDB unnecessarily reads the page even though it has fully initialized
      buffered redo log records. Allow the page initialization redo log to
      apply for the page in buf_page_get_gen() during recovery.
      - Renamed buf_page_get_gen() to buf_page_get_low()
      - Newly added buf_page_get_gen() will check for buffered redo log for
      the particular page id during recovery
      - Added new function buf_page_mtr_lock() which basically latches the page
      for the given latch type.
      - recv_recovery_create_page() is inline function which creates a page
      if it has page initialization redo log records.
      6697135c
  21. 21 Mar, 2020 2 commits
  22. 20 Mar, 2020 2 commits
  23. 19 Mar, 2020 1 commit
    • Marko Mäkelä's avatar
      MDEV-13626: Clean up the buffer pool resizing tests from MySQL 5.7 · 9fd692ae
      Marko Mäkelä authored
      buf_pool_resize(): Simplify the fault injection
      for innodb.buf_pool_resize_oom.
      
      innodb.buf_pool_resize_oom: Use a small buffer pool.
      
      innodb.innodb_buffer_pool_load_now: Make use of the sequence engine,
      to avoid creating explicit InnoDB record locks. Clean up the
      accesses to information_schema.innodb_buffer_page_lru.
      9fd692ae
  24. 12 Mar, 2020 1 commit
  25. 31 Jan, 2020 1 commit
  26. 23 Dec, 2019 1 commit
  27. 18 Dec, 2019 2 commits
  28. 09 Oct, 2019 1 commit
  29. 24 Jul, 2019 1 commit
  30. 11 Jul, 2019 1 commit
    • Thirunarayanan Balathandayuthapani's avatar
      MDEV-19978 Page read from tablespace is corrupted · aba2b41e
      Thirunarayanan Balathandayuthapani authored
      Problem:
      =======
        Checksum fields can have value as zero. In that case, InnoDB falsely
      consider that page should be all zeroes. It leads to wrong detection of page
      corruption.
      
      Solution:
      ========
      	Remove the condition that checks if checksum fields are zero then
      page should be all zeroes.
      aba2b41e
  31. 01 Jul, 2019 1 commit
    • Marko Mäkelä's avatar
      MDEV-19766: Disable page dump output in debug builds · 5a136d84
      Marko Mäkelä authored
      The test innodb.leaf_page_corrupted_during_recovery
      fails on buildbot with
      
      Warning	1406	Data too long for column 'line' at row 10
      line
       len 16384; hex ...
      
      because of a page dumps that InnoDB is generating for a corrupted page
      
      Since this test is using debug instrumentation, we will solve the
      issue by disabling page dumps in debug builds altogether. Users of
      debug builds will likely know how to extract page dumps in other means.
      
      Page dump output could sometimes be useful when diagnosing problems
      that users are facing. Hence we will keep the page dump output in
      non-debug (release) builds.
      5a136d84
  32. 28 Jun, 2019 1 commit
    • Thirunarayanan Balathandayuthapani's avatar
      MDEV-17228 Encrypted temporary tables are not encrypted · 723a4b1d
      Thirunarayanan Balathandayuthapani authored
      - Introduce a new variable called innodb_encrypt_temporary_tables which is
      a boolean variable. It decides whether to encrypt the temporary tablespace.
      - Encrypts the temporary tablespace based on full checksum format.
      - Introduced a new counter to track encrypted and decrypted temporary
      tablespace pages.
      - Warnings issued if temporary table creation has conflict value with
      innodb_encrypt_temporary_tables
      - Added a new test case which reads and writes the pages from/to temporary
      tablespace.
      723a4b1d
  33. 13 Jun, 2019 1 commit
    • Thirunarayanan Balathandayuthapani's avatar
      MDEV-19435 buf_fix_count > 0 for corrupted page when it exits the LRU list · e9145aab
      Thirunarayanan Balathandayuthapani authored
      Problem:
      =========
      One of the purge thread access the corrupted page and tries to remove from
      LRU list. In the mean time, other purge threads are waiting for same page
      in buf_wait_for_read(). Assertion(buf_fix_count == 0) fails for the
      purge thread which tries to remove the page from LRU list.
      
      Solution:
      ========
      - Set the page id as FIL_NULL to indicate the page is corrupted before
      removing the block from LRU list. Acquire hash lock for the particular
      page id and wait for the other threads to release buf_fix_count
      for the block.
      
      - Added the error check for btr_cur_open() in row_search_on_row_ref().
      e9145aab
  34. 05 Jun, 2019 1 commit
  35. 28 May, 2019 1 commit
  36. 11 May, 2019 1 commit