1. 31 Jul, 2020 6 commits
    • Thirunarayanan Balathandayuthapani's avatar
      MDEV-14711 Fix-up · 5ec40fbb
      Thirunarayanan Balathandayuthapani authored
      5ec40fbb
    • Thirunarayanan Balathandayuthapani's avatar
      MDEV-22511 innodb.truncate_foreign failed in buildbot with wrong error code · a6066e23
      Thirunarayanan Balathandayuthapani authored
      - Adding lock_wait_timeout value as 1 make sure that truncate table
      fails instead of making MDL timeout.
      a6066e23
    • Marko Mäkelä's avatar
      MDEV-11799 Doublewrite recovery can corrupt data pages · 879ba197
      Marko Mäkelä authored
      The purpose of the InnoDB doublewrite buffer is to make InnoDB
      tolerant against cases where the server was killed in the middle
      of a page write. (In Linux, killing a process may interrupt a
      write system call, typically on a 4096-byte boundary.)
      
      There may exist multiple copies of a page number in the doublewrite
      buffer. Recovery should choose the latest valid copy of the page.
      By design, the FIL_PAGE_LSN must not precede the latest checkpoint LSN
      nor be later than the end of the recovered log.
      
      For page_compressed and encrypted pages, we were missing proper
      consistency checks. In the 10.4 data set generated for in MDEV-23231,
      the data file contained a valid page_compressed page, and an
      identical copy of that page was also present in the doublewrite
      buffer. But, recovery would incorrectly consider the page invalid
      and restore an uncompressed copy of the same page that had been
      written before the log checkpoint. (In fact, no redo log was to
      be applied to that page.)
      
      buf_dblwr_process(): Validate the FIL_PAGE_LSN in the doublewrite
      buffer pages, and always skip page 0, because those pages should
      have been recovered by Datafile::restore_from_doublewrite() if
      necessary.
      
      Datafile::restore_from_doublewrite(): Choose the latest applicable
      page from the doublewrite buffer.
      
      recv_dblwr_t::find_page(): Also validate encrypted or
      page_compressed pages.
      
      recv_dblwr_t::validate_page(): New function to validate a page,
      either a copy in a data file or in the doublewrite buffer.
      Also validate encrypted or page_compressed pages.
      
      This is joint work with Thirunarayanan Balathandayuthapani.
      879ba197
    • Marko Mäkelä's avatar
      MDEV-23198 Crash in REPLACE · f35d1721
      Marko Mäkelä authored
      row_vers_impl_x_locked_low(): clust_offsets may point to memory
      that is allocated by mem_heap_alloc() and may have been freed.
      For initializing clust_offsets, try to use the stack-allocated
      buffer instead of a pointer that may point to freed memory.
      
      This fixes a regression that was introduced in
      commit f0aa073f (MDEV-20950).
      f35d1721
    • Nikita Malyavin's avatar
      MDEV-18042 Server crashes upon adding a non-null date column under... · fd0abc89
      Nikita Malyavin authored
      MDEV-18042 Server crashes upon adding a non-null date column under NO_ZERO_DATE with ALGORITHM=INPLACE
      
      accept table_name and db_name instead of table_share in make_truncated_value_warning
      fd0abc89
    • Nikita Malyavin's avatar
      MDEV-19338 InnoDB: Failing assertion: !cursor->index->is_committed() · 91ebf184
      Nikita Malyavin authored
      Call mark_columns_per_binlog_row_image before find_row() to set up table->vcol_set early,
      so the virtual column value will be updated after record read (ha_rnd_pos/ha_index_next/etc)
      by table->update_virtual_fields() call
      91ebf184
  2. 30 Jul, 2020 2 commits
    • Thirunarayanan Balathandayuthapani's avatar
      MDEV-23332 Index online status assert failure in btr_search_drop_page_hash_index · 8a612314
      Thirunarayanan Balathandayuthapani authored
      Problem:
      ========
      In row_merge_drop_indexes(), InnoDB drops only the index from
      dictionary and frees the index pages but it maintains the index
      object if the table is being used by other DML threads. It sets
      the online status of the index to ONLINE_INDEX_ABORTED_DROPPED.
      Removing the index from dictionary doesn't remove the
      corressponding ahi entries of the index. When block is being
      reused, InnoDB tries to remove ahi entries for the block and
      it fails if index online status is ONLINE_INDEX_ABORTED_DROPPED.
      
      Fix:
      ====
      MDEV-22456 allows the index ahi entries to be dropped lazily.
      so checking online status in btr_search_drop_page_hash_index()
      is meaningless and should be removed.
      8a612314
    • Marko Mäkelä's avatar
      MDEV-23339 innodb_force_recovery=2 may still abort the rollback of recovered transactions · c5d4dd25
      Marko Mäkelä authored
      trx_rollback_active(), trx_rollback_resurrected(): Replace
      an incorrect condition that we failed to replace in
      commit b68f1d84 (MDEV-21217).
      c5d4dd25
  3. 29 Jul, 2020 2 commits
  4. 28 Jul, 2020 7 commits
  5. 27 Jul, 2020 4 commits
    • Daniel Black's avatar
      MDEV-12474: rocksdb: mtr - rocksdb.concurrent_alter use sh · 186d9d0d
      Daniel Black authored
      FreeBSD doesn't have bash installed by default and sh
      has sufficient job control for this test.
      
      $  mysql-test/mtr --mem --max-test-fail=30 --force --parallel=1 rocksdb.concurrent_alter
      Logging: /home/dan/mariadb-server-10.5/mysql-test/mysql-test-run.pl  --mem --max-test-fail=30 --force --parallel=1 rocksdb.concurrent_alter
      vardir: /usr/home/dan/build-mariadb-server-10.5/mysql-test/var
      Checking leftover processes...
      Removing old var directory...
      Creating var directory '/usr/home/dan/build-mariadb-server-10.5/mysql-test/var'...
       - symlinking 'var' to '/tmp/var_auto_P81m'
      Checking supported features...
      MariaDB Version 10.5.4-MariaDB
       - SSL connections supported
      
       - binaries built with wsrep patch
      Collecting tests...
      Installing system database...
      
      ==============================================================================
      
      TEST                                      RESULT   TIME (ms) or COMMENT
      --------------------------------------------------------------------------
      
      worker[1] Using MTR_BUILD_THREAD 300, with reserved ports 16000..16019
      rocksdb.concurrent_alter 'write_committed' [ pass ]  16348
      rocksdb.concurrent_alter 'write_prepared' [ pass ]  16771
      --------------------------------------------------------------------------
      The servers were restarted 1 times
      Spent 33.119 of 41 seconds executing testcases
      
      Completed: All 2 tests were successful.
      
      $ uname -a
      FreeBSD freebsd 12.1-RELEASE-p6 FreeBSD 12.1-RELEASE-p6 GENERIC  amd64
      186d9d0d
    • Thirunarayanan Balathandayuthapani's avatar
      MDEV-23233: Race condition for btr_search_drop_page_hash_index() in buf_page_create() · a1f899a8
      Thirunarayanan Balathandayuthapani authored
      commit ad6171b9 (MDEV-22456)
      introduced code to buf_page_create() that would lazily drop
      adaptive hash index entries for an index that has been
      evicted from the data dictionary cache.
      
      Unfortunately, that call was missing adequate protection.
      While the btr_search_drop_page_hash_index(block) was executing,
      the block could be reused for something else.
      
      buf_page_create(): If btr_search_drop_page_hash_index() must be
      invoked, pin the block before releasing the buf_pool->page_hash lock,
      so that the block cannot be grabbed by other threads.
      a1f899a8
    • Jan Lindström's avatar
      Enable fixed test case. · b4c74210
      Jan Lindström authored
      b4c74210
    • Varun Gupta's avatar
      MDEV-18916: crash in Window_spec::print_partition() with decimals · a6410deb
      Varun Gupta authored
      Removed an unnecessary ifndef which was printing the window name for a named
      window only in the case of debug build. The print() for the window function
      should behave in the same way on both release and debug builds.
      a6410deb
  6. 24 Jul, 2020 4 commits
    • Thirunarayanan Balathandayuthapani's avatar
      MDEV-14711 Assertion `mode == 16 || mode == 12 ||... · 5f1ec5cb
      Thirunarayanan Balathandayuthapani authored
      MDEV-14711 Assertion `mode == 16 || mode == 12 || !fix_block->page.file_page_was_freed' failed in buf_page_get_gen (rollback requesting a freed undo page)
      
      Problem:
      =======
      In buf_cur_optimistic_latch_leaves(), requesting a left block with BTR_GET
      after releasing current block. But there is no guarantee that left block
      could be still available.
      
      Fix:
      ====
      
      (1) In btr_cur_optimistic_latch_leaves(), replace the BUF_GET with
      BUF_GET_POSSIBLY_FREED for fetching left block.
      (2) Once InnoDB acquires left block, it should check FIL_PAGE_NEXT with
      current block page number. If not, release cursor->left_block and return
      false.
      5f1ec5cb
    • mkaruza's avatar
      MDEV-15236: galera_ist_progress fails when trying to read transfer status · a3a249c7
      mkaruza authored
      Fixed by new  recording test recording
      a3a249c7
    • sjaakola's avatar
      MDEV-20928 mtr test galera.galera_var_innodb_disallow_writes test failure · c9928cc0
      sjaakola authored
      The sporadic test hangs happen because of mutex dealock between innodb
      background threads and two test connection executions.
      The test sets variable innodb_disallow_writes, which blocks all writes
      to filesyste. The test logic is to execute an INSERT, which should hang
      because of filesytstem writes are blocked, and through another session
      verify by SELECT that this hanging happens. The SELECT session will then
      release innodb_disallow_writes blocking.
      
      However, filesystem write  blocking affects also innodb background threads
      and they may hang while keeping some other resources locked.
      As an example, in one test hang situation, buffer pool access was blocked.
      And, if buffer pool is blocked, the test connections will be blocked as well,
      and the SELECT session will not be able to continue to release the
      innodb_disallow_writes.
      
      The fix in this commit is refactoring of the test logic.
      The test will now set first innodb_disallow_writes blocking, and then record
      a hash of data directory's filesystem contents. This works as checksum of the
      state of data on the datadirectory.
      
      Then some SQL load is tried on both nodes, these sessions will be blocking
      due to frozen file system state. The test will have a short sleep to allow
      innodb background threads to loop and possibly encounter innodb_disallow_writes
      blocking as well.
      
      After the sleep, the test will record file system checksun for the second time,
      and then release the innodb_disallow-writes blocking.
      
      Finally, the two checksums are compared, they should be identical to verify that
      nothing was written on datadirectory during the test execution.
      
      The checksum is implemented by md5sum hash over all files found in datadirectory
      by find command. all these file hashes are hashed together by one more md5sum.
      
      The test therefore depends on md5sum and find. find may work differently with some
      OS distributions, e.g. freebsd may be problematic.
      c9928cc0
    • Jan Lindström's avatar
      MDEV-18177 : Galera test failure on galera_autoinc_sst_mariabackup · ba23e6d7
      Jan Lindström authored
      Add wait_condition
      ba23e6d7
  7. 23 Jul, 2020 3 commits
    • Thirunarayanan Balathandayuthapani's avatar
      MDEV-23134 SEGV in dict_load_table_one during restart after server crash · b3b1c51e
      Thirunarayanan Balathandayuthapani authored
      Problem:
      ========
      dict_load_table_one() doesn't handle the scenario where clustered
      index page is FIL_NULL when DICT_ERR_IGNORE_INDEX_ROOT mode
      is set.
      
      Fix:
      ====
      InnoDB should set the file_unreadable when it can't find the
      clustered index root page.
      b3b1c51e
    • Thirunarayanan Balathandayuthapani's avatar
      MDEV-20638 Remove the deadcode from srv_master_thread() and srv_active_wake_master_thread_low() · fe39d02f
      Thirunarayanan Balathandayuthapani authored
      - Due to commit fe95cb2e (MDEV-16125),
      InnoDB master thread does not need to call srv_resume_thread()
      and therefore there is no need to wake up the thread.
      Due to the above patch, InnoDB should remove the following dead code.
      
      srv_check_activity(): Makes the parameter as in,out and returns the
      recent activity value
      
      innobase_active_small(): Removed
      
      srv_active_wake_master_thread(): Removed
      
      srv_wake_master_thread(): Removed
      
      srv_active_wake_master_thread_low(): Removed
      
      Simplify srv_master_thread() and remove switch cases, added the assert.
      
      Replace srv_wake_master_thread() with srv_inc_activity_count()
      
      INNOBASE_WAKE_INTERVAL: Removed
      fe39d02f
    • Sujatha's avatar
      MDEV-14203: rpl.rpl_extra_col_master_myisam,... · b3dd95e0
      Sujatha authored
      MDEV-14203: rpl.rpl_extra_col_master_myisam, rpl.rpl_slave_load_tmpdir_not_exist failed in buildbot with a warning
      
      Problem:
      =======
      rpl.rpl_slave_load_tmpdir_not_exist 'stmt' w3 [ fail ]  Found warnings/errors
      in server log file!
      
      Test ended at 2017-09-27 20:34:55
      [Warning] Master is configured to log replication events with checksum, but
      will not send such events to slaves that cannot process them
      ^ Found warnings in /mnt/buildbot/build/mariadb-10.2.10/mysql-test/var/3/log/mysqld.1.err
      ok
      Analysis:
      ========
      When slave tries to connect to master 'get_master_version_and_clock' function
      is invoked to perform elaborated slave-master handshake. During this process
      slave server queries master server, to know if it is checksum aware and at the
      same time master is notified about its CRC-awareness. The master's side
      instant value of @@global.binlog_checksum is stored in the dump thread's
      uservar area as well as cached locally to become known in consensus by master
      and slave.
      
      Post hand-shake slave requests master for binlog dump. It sends
      'COM_BINLOG_DUMP'. This command is sent to master by 'cli_advanced_command'
      call. If there is some temporary network failure during this request_dump
      call, 'end_server' is invoked to close the current connection between master
      and slave. Upon connection close the dump thread on the master gets terminated
      and it clears the 'uservar' data it got through master-slave handshake.
      
      The 'COM_BINLOG_DUMP' command is sent once again without master-slave
      handshake. Since the checksum data is not available with new dump thread a
      warning gets reported.
      
      Fix:
      ===
      Upon network write error donot attempt reconnect, proceed to master-slave
      handshake. This ensures that master is aware of slave's capability to use
      checksums.
      b3dd95e0
  8. 22 Jul, 2020 3 commits
  9. 21 Jul, 2020 2 commits
    • Nikita Malyavin's avatar
      fix c++98 build · ebca70ea
      Nikita Malyavin authored
      ebca70ea
    • Nikita Malyavin's avatar
      MDEV-16039 Crash when selecting virtual columns generated using functions with DAYNAME() · 5acd391e
      Nikita Malyavin authored
      * Allocate items on thd->mem_root while refixing vcol exprs
      * Make vcol tree changes register and roll them back after the statement is executed.
      
      Explanation:
      Due to collation implementation specifics an Item tree could change while fixing.
      The tricky thing here is to make it on a proper arena.
      It's usually not a problem when a field is deterministic, however, makes a pain vice-versa, during allocation allocating.
      A non-deterministic field should be refixed on each statement, since it depends on the environment state.
      Changing the tree will be temporary and therefore it should be reverted after the statement execution.
      5acd391e
  10. 20 Jul, 2020 4 commits
    • Marko Mäkelä's avatar
      Merge 10.1 into 10.2 · ca9276e3
      Marko Mäkelä authored
      ca9276e3
    • Marko Mäkelä's avatar
      MDEV-23190 InnoDB data file extension is not crash-safe · 57ec42bc
      Marko Mäkelä authored
      When InnoDB is extending a data file, it is updating the FSP_SIZE
      field in the first page of the data file.
      
      In commit 8451e090 (MDEV-11556)
      we removed a work-around for this bug and made recovery stricter,
      by making it track changes to FSP_SIZE via redo log records, and
      extend the data files before any changes are being applied to them.
      
      It turns out that the function fsp_fill_free_list() is not crash-safe
      with respect to this when it is initializing the change buffer bitmap
      page (page 1, or generally, N*innodb_page_size+1). It uses a separate
      mini-transaction that is committed (and will be written to the redo
      log file) before the mini-transaction that actually extended the data
      file. Hence, recovery can observe a reference to a page that is
      beyond the current end of the data file.
      
      fsp_fill_free_list(): Initialize the change buffer bitmap page in
      the same mini-transaction.
      
      The rest of the changes are fixing a bug that the use of the separate
      mini-transaction was attempting to work around. Namely, we must ensure
      that no other thread will access the change buffer bitmap page before
      our mini-transaction has been committed and all page latches have been
      released.
      
      That is, for read-ahead as well as neighbour flushing, we must avoid
      accessing pages that might not yet be durably part of the tablespace.
      
      fil_space_t::committed_size: The size of the tablespace
      as persisted by mtr_commit().
      
      fil_space_t::max_page_number_for_io(): Limit the highest page
      number for I/O batches to committed_size.
      
      MTR_MEMO_SPACE_X_LOCK: Replaces MTR_MEMO_X_LOCK for fil_space_t::latch.
      
      mtr_x_space_lock(): Replaces mtr_x_lock() for fil_space_t::latch.
      
      mtr_memo_slot_release_func(): When releasing MTR_MEMO_SPACE_X_LOCK,
      copy space->size to space->committed_size. In this way, read-ahead
      or flushing will never be invoked on pages that do not yet exist
      according to FSP_SIZE.
      57ec42bc
    • Marko Mäkelä's avatar
      98e2c17e
    • Marko Mäkelä's avatar
      14543afd
  11. 18 Jul, 2020 1 commit
  12. 16 Jul, 2020 2 commits