1. 16 Jun, 2021 2 commits
    • Marko Mäkelä's avatar
      MDEV-25936 Crash during DDL that involves FULLTEXT INDEX · da65cb4d
      Marko Mäkelä authored
      In commit 1bd681c8 we introduced
      a work-around for the missing MDL protection when the internal
      tables of FULLTEXT INDEX are dropped during DDL operations.
      That work-around suffered from a race condition. A purge thread
      could have narrowly passed purge_sys.check_stop_FTS() and then
      (while holding dict_sys.mutex) acquire a table reference right
      after fts_lock_table() determined that no references were being
      held.
      
      fts_lock_table(): Protect the reference check with dict_sys.mutex.
      
      Thanks to Thirunarayanan Balathandayuthapani for repeating the
      failure and testing the fix.
      da65cb4d
    • Marko Mäkelä's avatar
      MDEV-25910: Aim to make all InnoDB DDL durable · 71964c76
      Marko Mäkelä authored
      Before any committed DDL operation returns control to the caller,
      we must ensure that it will have been durably committed before the
      ddl_log state may be changed, no matter if
      innodb_flush_log_at_trx_commit=0 is being used.
      
      Operations that would involve deleting files were already safe,
      because the durable write of the FILE_DELETE record that would precede
      the file deletion would also have made the commit durable.
      
      Furthermore, when we clean up ADD INDEX stubs that were left behind
      by a previous ha_innobase::commit_inplace_alter_table(commit=false)
      when MDL could not be acquired, we will use the same interface as
      for dropping indexes. This safety measure should be dead code,
      because ADD FULLTEXT INDEX is not supported online, and dropping indexes
      only involves file deletion for FULLTEXT INDEX.
      71964c76
  2. 15 Jun, 2021 3 commits
    • Marko Mäkelä's avatar
      MDEV-25910: Make ALTER TABLE...ALGORITHM=COPY durable again · e5b9dc15
      Marko Mäkelä authored
      ha_innobase::extra(HA_EXTRA_END_ALTER_COPY): Flush the redo log
      because the main transaction will not have written any log and
      thus will not initiate any log write either. But the DDL recovery
      layer expects that the operation will have been committed.
      e5b9dc15
    • Sergei Golubchik's avatar
      update C/C to 3.2.2-rc · e0647dc7
      Sergei Golubchik authored
      e0647dc7
    • Andrei Elkin's avatar
      MDEV-21117 post-push fixes · 79a2dbc8
      Andrei Elkin authored
      1. work around MDEV-25912 to not apply assert
         at wsrep running time;
      2. handle wsrep mode of the server recovery
      3. convert hton calls to static binlog_commit ones.
      4. satisfy MSAN complain on uninitialized std::pair
      79a2dbc8
  3. 14 Jun, 2021 19 commits
  4. 11 Jun, 2021 11 commits
    • Sujatha's avatar
      MDEV-21117: refine the server binlog-based recovery for semisync · 6c39eaeb
      Sujatha authored
      Problem:
      =======
      When the semisync master is crashed and restarted as slave it could
      recover transactions that former slaves may never have seen.
      A known method existed to clear out all prepared transactions
      with --tc-heuristic-recover=rollback does not care to adjust
      binlog accordingly.
      
      Fix:
      ===
      The binlog-based recovery is made to concern of the slave semisync role of
      post-crash restarted server.
      No changes in behavior is done to the "normal" binloggging server
      and the semisync master.
      
      When the restarted server is configured with
        --rpl-semi-sync-slave-enabled=1
      the refined recovery attempts to roll back prepared transactions
      and truncate binlog accordingly.
      In case of a partially committed (that is committed at least
      in one of the engine participants) such transaction gets committed.
      It's guaranteed no (partially as well) committed transactions
      exist beyond the truncate position.
      In case there exists a non-transactional replication event
      (being in a way a committed transaction) past the
      computed truncate position the recovery ends with an error.
      
      As after master crash and failover to slave, the demoted-to-slave
      ex-master must be ready to face and accept its own (generated by)
      events, without generally necessary --replicate-same-server-id.
      So the acceptance conditions are relaxed for the semisync slave
      to accept own events without that option.
      While gtid_strict_mode ON ensures no duplicate transaction can be
      (re-)executed the master_use_gtid=none slave has to be
      configured with --replicate-same-server-id.
      
      *NOTE* for reviewers.
      
      This patch does not handle the user XA which is done
      in next git commit.
      6c39eaeb
    • Marko Mäkelä's avatar
      82c07b17
    • Sergei Golubchik's avatar
      cleanup perfschema.short_options_1 test · f4943b4a
      Sergei Golubchik authored
      the test tests whether short options work on the server command line
      
      * remove 'show variables' for variables not affected by short options
      * remove options, that are not short
      * remove options, that cannot be tested from SQL
        * in particular, -T12 doesn't affect the test output,
          but cases ~30sec delay on shutdown
      * use -W1 as -W2 is the default, so doesn't affect the test output
      f4943b4a
    • Sergei Golubchik's avatar
      773ee80f
    • Sergei Golubchik's avatar
      don't show DBUG_ASSERT to plugins · 4d0b3395
      Sergei Golubchik authored
      4d0b3395
    • Sergei Golubchik's avatar
      cleanup: rename Protocol::store() to Protocol::store_datetime() · abc3889f
      Sergei Golubchik authored
      to match the naming pattern of all other Protocol::store_xxx() methods
      abc3889f
    • Sergei Golubchik's avatar
      cleanup: Field_set::empty_set_string · 59b51e6a
      Sergei Golubchik authored
      in particular, it overwrites pre-allocated buffer in val_buffer,
      so following val_buffer->append()'s cause totally unnecessary
      mallocs.
      59b51e6a
    • Sergei Golubchik's avatar
      cleanup: formatting · 3648b333
      Sergei Golubchik authored
      also avoid an oxymoron of using `MYSQL_PLUGIN_IMPORT` under
      `#ifdef MYSQL_SERVER`, and empty_clex_str is so trivial that a plugin
      can define it if needed.
      3648b333
    • Sergei Golubchik's avatar
      change maturity to gamma · 89342a3b
      Sergei Golubchik authored
      89342a3b
    • Marko Mäkelä's avatar
      MDEV-25288 follow-up: Remove mysql-test/unstable-tests · ff3fd022
      Marko Mäkelä authored
      We no longer maintain the unstable-tests collection.
      In commit 3635280c we added
      a smoke test collection, which independent MariaDB package maintainers
      can expect to pass.
      ff3fd022
    • Krunal Bauskar's avatar
      MDEV-25882: Statistics used to track b-tree (non-adaptive) searches · 102ff420
      Krunal Bauskar authored
                  should be updated only when adaptive hashing is turned-on
      
      Currently, btr_cur_n_non_sea is used to track the search that missed
      adaptive hash index. adaptive hash index is turned off by default
      but the said variable is updated always though the value of it makes sense
      only when an adaptive index is enabled. It is meant to check how many
      searches didn't go through an adaptive hash index.
      
      Given a global variable that is updated on each search path it causes
      a contention with a multi-threaded workload.
      
      Patch moves the said variables inside a loop that is now updated
      only when the adaptive hash index is enabled and that in theory should
      also, reduce the update frequency of the said variable as the majority of
      the request should be serviced through the adaptive hash index.
      
      Variables (btr_cur_n_non_sea and btr_cur_n_sea) are also converted to
      use distributed counter to avoid contention.
      
      User visible changes:
      
      This also means that user will now see
      Innodb_adaptive_hash_non_hash_searches (viewed as part of show status)
      only if code is compiled with DWITH_INNODB_AHI=ON (default) and it will
      be updated only if innodb_adaptive_hash_index=1 else it reported as 0.
      102ff420
  5. 09 Jun, 2021 5 commits
    • Marko Mäkelä's avatar
      MDEV-25750: Assertion wait_lock->is_waiting() in lock_wait_rpl_report · 6fbf978e
      Marko Mäkelä authored
      lock_wait_rpl_report(): Tolerate the loss of a lock wait.
      commit c68007d9 (MDEV-24738)
      had introduced this rare failure.
      6fbf978e
    • Marko Mäkelä's avatar
      MDEV-25506 (3 of 3): Do not delete .ibd files before commit · 1bd681c8
      Marko Mäkelä authored
      This is a complete rewrite of DROP TABLE, also as part of other DDL,
      such as ALTER TABLE, CREATE TABLE...SELECT, TRUNCATE TABLE.
      
      The background DROP TABLE queue hack is removed.
      If a transaction needs to drop and create a table by the same name
      (like TRUNCATE TABLE does), it must first rename the table to an
      internal #sql-ib name. No committed version of the data dictionary
      will include any #sql-ib tables, because whenever a transaction
      renames a table to a #sql-ib name, it will also drop that table.
      Either the rename will be rolled back, or the drop will be committed.
      
      Data files will be unlinked after the transaction has been committed
      and a FILE_RENAME record has been durably written. The file will
      actually be deleted when the detached file handle returned by
      fil_delete_tablespace() will be closed, after the latches have been
      released. It is possible that a purge of the delete of the SYS_INDEXES
      record for the clustered index will execute fil_delete_tablespace()
      concurrently with the DDL transaction. In that case, the thread that
      arrives later will wait for the other thread to finish.
      
      HTON_TRUNCATE_REQUIRES_EXCLUSIVE_USE: A new handler flag.
      ha_innobase::truncate() now requires that all other references to
      the table be released in advance. This was implemented by Monty.
      
      ha_innobase::delete_table(): If CREATE TABLE..SELECT is detected,
      we will "hijack" the current transaction, drop the table in
      the current transaction and commit the current transaction.
      This essentially fixes MDEV-21602. There is a FIXME comment about
      making the check less failure-prone.
      
      ha_innobase::truncate(), ha_innobase::delete_table():
      Implement a fast path for temporary tables. We will no longer allow
      temporary tables to use the adaptive hash index.
      
      dict_table_t::mdl_name: The original table name for the purpose of
      acquiring MDL in purge, to prevent a race condition between a
      DDL transaction that is dropping a table, and purge processing
      undo log records of DML that had executed before the DDL operation.
      For #sql-backup- tables during ALTER TABLE...ALGORITHM=COPY, the
      dict_table_t::mdl_name will differ from dict_table_t::name.
      
      dict_table_t::parse_name(): Use mdl_name instead of name.
      
      dict_table_rename_in_cache(): Update mdl_name.
      
      For the internal FTS_ tables of FULLTEXT INDEX, purge would
      acquire MDL on the FTS_ table name, but not on the main table,
      and therefore it would be able to run concurrently with a
      DDL transaction that is dropping the table. Previously, the
      DROP TABLE queue hack prevented a race between purge and DDL.
      For now, we introduce purge_sys.stop_FTS() to prevent purge from
      opening any table, while a DDL transaction that may drop FTS_
      tables is in progress. The function fts_lock_table(), which will
      be invoked before the dictionary is locked, will wait for
      purge to release any table handles.
      
      trx_t::drop_table_statistics(): Drop statistics for the table.
      This replaces dict_stats_drop_index(). We will drop or rename
      persistent statistics atomically as part of DDL transactions.
      On lock conflict for dropping statistics, we will fail instantly
      with DB_LOCK_WAIT_TIMEOUT, because we will be holding the
      exclusive data dictionary latch.
      
      trx_t::commit_cleanup(): Separated from trx_t::commit_in_memory().
      Relax an assertion around fts_commit() and allow DB_LOCK_WAIT_TIMEOUT
      in addition to DB_DUPLICATE_KEY. The call to fts_commit() is
      entirely misplaced here and may obviously break the consistency
      of transactions that affect FULLTEXT INDEX. It needs to be fixed
      separately.
      
      dict_table_t::n_foreign_key_checks_running: Remove (MDEV-21175).
      The counter was a work-around for missing meta-data locking (MDL)
      on the SQL layer, and not really needed in MariaDB.
      
      ER_TABLE_IN_FK_CHECK: Replaced with ER_UNUSED_28.
      
      HA_ERR_TABLE_IN_FK_CHECK: Remove.
      
      row_ins_check_foreign_constraints(): Do not acquire
      dict_sys.latch either. The SQL-layer MDL will protect us.
      
      This was reviewed by Thirunarayanan Balathandayuthapani
      and tested by Matthias Leich.
      1bd681c8
    • Marko Mäkelä's avatar
      MDEV-25180 fixup: Assertion table->def_trx_id == trx->id... failed in row_merge_drop_indexes() · 3f78fbc5
      Marko Mäkelä authored
      We must invoke online_retry_drop_indexes() before
      ha_innobase_inplace_ctx::trx->id is assigned.
      3f78fbc5
    • Marko Mäkelä's avatar
      MDEV-25852: Orphan #sql*.ibd files are left behind · 6a4e5bf1
      Marko Mäkelä authored
      The implementation of MDEV-24626 was not entirely correct.
      We could occasionally fail to remove some *.ibd files on recovery.
      
      deferred_spaces: Keep track of FILE_DELETE records.
      
      deferred_spaces.add(): Do not allow duplicate file names.
      
      recv_rename_files(): Preserve some of renamed_spaces entries for
      deferred_spaces.reinit_all().
      
      Thanks to Thirunarayanan Balathandayuthapani for noticing that
      deferred_spaces.add() must filter out duplicate file names,
      as well as some debugging help.
      6a4e5bf1
    • Marko Mäkelä's avatar
      Merge 10.5 into 10.6 · 65f1a427
      Marko Mäkelä authored
      65f1a427