1. 11 Dec, 2018 8 commits
    • Sergei Golubchik's avatar
      mysqld: ignore SIGHUP sent by the kernel · 07e9b138
      Sergei Golubchik authored
      SIGHUP causes debug info in the error log and reload of
      logs/privileges/tables/etc. The server should only do it when
      a user intentionally sends SUGHUP, not when a parent terminal gets
      disconnected or something.
      
      In particular, not ignoring kernel SIGHUP causes FLUSH PRIVILEGES
      at some random point during non-systemd Debian upgrades (Debian
      restarts mysqld, debian-start script runs mysql_upgrade in the background,
      postinit script ends and kernel sends SIGHUP to all background processes
      it has started). And during mysql_upgrade privilege tables aren't
      necessarily ready to be reloaded.
      07e9b138
    • Sergei Golubchik's avatar
      simplify usage of logger.set_handlers() · 97dbb356
      Sergei Golubchik authored
      97dbb356
    • Sergei Golubchik's avatar
      more tests for mysql_install_db.exe · a4ac987c
      Sergei Golubchik authored
      a4ac987c
    • Sergei Golubchik's avatar
      MDEV-17926 FederatedX TODO is obsolete · 7af62f8a
      Sergei Golubchik authored
      7af62f8a
    • Alexey Botchkov's avatar
      MDEV-5313 Improving audit api. · b1527ef5
      Alexey Botchkov authored
      Service added to handle json.
      b1527ef5
    • Marko Mäkelä's avatar
      MDEV-17901 Crash after instant DROP COLUMN of AUTO_INCREMENT column · dc6ad597
      Marko Mäkelä authored
      We failed to reset the dict_table_t::persistent_autoinc after
      instantly dropping an AUTO_INCREMENT column, causing a bogus
      call to row_parse_int() on a subsequent insert.
      dc6ad597
    • Jiaye Wu's avatar
      Fix UNICODE issue of dlerror() · 8dc460b8
      Jiaye Wu authored
      Current implementation is conflicting. If `UNICODE` is defined, `FormatMessage()` will be `FormatMessageW()`, and variable `win_errormsg` with type `char` can not be passed to it, which should be changed to `TCHAR` instead. Since we don't use UNICODE here, we can use `FormatMessageA()` directly to avoid conversion error.
      ```
      my_global.h(1092): error C2664: 'DWORD FormatMessageW(D
      WORD,LPCVOID,DWORD,DWORD,LPWSTR,DWORD,va_list *)' : cannot convert argument 5 from 'char [2048]' to 'LPWSTR'
      ```
      8dc460b8
    • Marko Mäkelä's avatar
      MDEV-17899: Fix a regression from MDEV-17793 · 1b31d885
      Marko Mäkelä authored
      The fix of MDEV-17793 was updating SYS_INDEXES.TABLE_ID in order
      to make the table invisible to purge (lazily delete old undo log
      records).
      
      By design of InnoDB, an update of TABLE_ID cannot be rolled back,
      because the rollback would effectively drop all indexes of the table
      due to the internal 'trigger' on SYS_INDEXES modifications.
      
      So, we revert the code change of MDEV-17793 and instead fix
      MDEV-17793 in a different way: by tweaking the undo log parsing
      during purge.
      
      The MDEV-17793 bug scenario is that a table becomes empty and
      a third instant ALTER TABLE is executed before purge processes
      the undo log record for the second instant ALTER TABLE. After
      this point, when purge sees the record, the table could have
      a mismatching number of rows.
      
      The test case works with this alternative fix. But what about
      a scenario where a fourth instant ALTER TABLE arrives before
      purge processes the second one? Could anything bad happen?
      
      Purge is only doing two things: First, free any BLOBs that
      were affected by the update record, and then, reset the
      DB_TRX_ID,DB_ROLL_PTR if a matching record is found.
      For the hidden metadata record, the only BLOB that we update
      is the hidden metadata BLOB that was introduced by MDEV-15562.
      Any other BLOBs (for the initial default values of instantly
      added columns) are never updated.
      
      So, in our scenario, the metadata BLOB that was created by
      the first instant ALTER TABLE (if it involved dropping or
      permuting columns) would be freed by purge when it is processing
      the undo record of the second ALTER TABLE. The BLOB value that
      was written by the second ALTER TABLE should be freed when
      the table is emptied. This is currently not done: MDEV-17383
      should fix that. There is no possibility of double-free, because
      purge would only free old values of BLOBs.
      
      What about MVCC and other callers of trx_undo_update_rec_get_update()?
      The answer is simple: they should never be accessing the hidden
      metadata record in the first place.
      
      dict_table_t::reassign_id(): Remove.
      
      btr_cur_pessimistic_delete(): Clarify a comment.
      
      row_mysql_table_id_reassign(), row_discard_tablespace_for_mysql():
      Add comments explaining that the operation cannot be rolled back.
      
      trx_undo_update_rec_get_update(): Avoid out-of-bounds access when
      parsing a metadata record. Avoid unnecessary memory allocation when
      filtering out fields from the update vector.
      1b31d885
  2. 10 Dec, 2018 12 commits
    • Sergey Vojtovich's avatar
      Fixed sporadic main.mdl_sync failure · 1c53aeff
      Sergey Vojtovich authored
      When update finishes, MDL_context::release_transactional_locks() first
      releases MDL_STATEMENT locks (MDL_BACKUP_DML and MDL_BACKUP_TRANS_DML
      in this particular cases), which allows FTWRL connection to proceed.
      Then MDL_TRANSACTION locks get released (MDL_SHARED_WRITE in this case).
      metadata_lock_info query may sporadically hit this gap and observe
      these otherwise unexpected locks.
      1c53aeff
    • Sergey Vojtovich's avatar
      Fixed main.create-big failure · 0fcb141f
      Sergey Vojtovich authored
      After MDEV-17772 table existence check is performed much earlier, so
      create_table_select_before_check_if_exists debug sync point is not
      reachable when table exists.
      
      Moved debug sync point to appropriate place.
      0fcb141f
    • Vladislav Vaintroub's avatar
      80491609
    • Alexander Barkov's avatar
    • Varun Gupta's avatar
      Updated the result for innodb.innodb_mysql · 5b3db871
      Varun Gupta authored
      The result is updated here because by changing the defaults for use_stat_tables
      EITs statistcs are providing more accurate statistics than INNODB statistics.
      5b3db871
    • Marko Mäkelä's avatar
      MDEV-17899 Assertion failures on rollback of instant ADD/DROP · f69879ba
      Marko Mäkelä authored
      row_undo_ins_remove_clust_rec(): When rolling back the insert
      of a metadata record, also reset the root page state within the
      same mini-transaction.
      
      dict_table_t::rollback_instant(): Do not allow index->n_core_fields
      to be larger than index->n_fields. This logic exists in the 10.3
      branch but was incorrectly removed as part of MDEV-15562 in 10.4.0.
      f69879ba
    • Vladislav Vaintroub's avatar
      Fix lock_ddl_per_table handling in mariabackup. · 3efed753
      Vladislav Vaintroub authored
      
      mariabackup.lock_ddl_per_table was broken, after MDEV-17772 got fixed.
      Fix removes MDL waiter killer, it is is not needed anymore.
      3efed753
    • Marko Mäkelä's avatar
      Merge 10.3 into 10.4 · fcb6bb4b
      Marko Mäkelä authored
      fcb6bb4b
    • Marko Mäkelä's avatar
      MDEV-17938: ALTER TABLE error handling accesses freed memory · a7251634
      Marko Mäkelä authored
      This regression was introduced in MDEV-11369 instant ADD COLUMN.
      
      prepare_inplace_alter_table_dict(): Avoid dereferencing ctx->new_table
      after row_create_table_for_mysql() returns a failure code, because
      it will have freed the memory.
      a7251634
    • Marko Mäkelä's avatar
      Merge 10.2 into 10.3 · 1d18665e
      Marko Mäkelä authored
      1d18665e
    • Marko Mäkelä's avatar
      MDEV-17938 ALTER TABLE reports ER_TABLESPACE_EXISTS after failed ALTER TABLE · 0d7cf06a
      Marko Mäkelä authored
      There was a race condition in the error handling of ALTER TABLE when
      the table contains FULLTEXT INDEX.
      
      During the error handling of an erroneous ALTER TABLE statement,
      when InnoDB would drop the internally created tables for FULLTEXT INDEX,
      it could happen that one of the hidden tables was being concurrently
      accessed by a background thread. Because of this, InnoDB would defer
      the drop operation to the background.
      
      However, related to MDEV-13564 backup-safe TRUNCATE TABLE and its
      prerequisite MDEV-14585, we had to make the background drop table queue
      crash-safe by renaming the table to a temporary name before enqueueing it.
      This renaming was introduced in a follow-up of the MDEV-13407 fix.
      As part of this rename operation, we were unnecessarily parsing the
      current SQL statement, because the same rename operation could also be
      executed as part of ALTER TABLE via ha_innobase::rename_table().
      
      If an ALTER TABLE statement was being refused due to incorrectly formed
      FOREIGN KEY constraint, then it could happen that the renaming of the hidden
      internal tables for FULLTEXT INDEX could also fail, triggering a host of
      error log messages, and causing a subsequent table-rebuilding ALTER TABLE
      operation to fail due to the tablespace already existing.
      
      innobase_rename_table(), row_rename_table_for_mysql(): Add the parameter
      use_fk for suppressing the parsing of FOREIGN KEY constraints. It
      will only be passed as use_fk=true by ha_innobase::rename_table(),
      which can be invoked as part of ALTER TABLE...ALGORITHM=COPY.
      0d7cf06a
    • Alexander Barkov's avatar
      Cleanup: Datetime() constructors accepting Longlong_hybrid/Sec6 do not need THD · e5144f4b
      Alexander Barkov authored
      Removing the unused THD* parameter.
      e5144f4b
  3. 09 Dec, 2018 20 commits
    • Sergey Vojtovich's avatar
      Moved early check for table existance to mysql_execute_command() · 8cf7e345
      Sergey Vojtovich authored
      MDEV-17772 - 3 way lock : ALTER, MDL, BACKUP STAGE BLOCK_DDL
      
      While waiting for a (potentially long) RO transaction or SELECT, DDL and
      LOCK TABLES ... WRITE hold protection against FTWRL and BACKUP STAGE.
      
      This effectively makes FTWRL/BACKUP STAGE indirectly wait for this RO
      transaction or SELECT to finish. Which is not great, as otherwise we
      could do something useful meanwhile.
      
      With this patch BACKUP lock is attempted to be acquired after TABLE/SCHEMA
      locks. If this attempt fails, TABLE/SCHEMA locks gets released and we
      start waiting for BACKUP lock. When wait finishes, BACKUP lock is released
      (to avoid deadlocks) and we attempt to acquire all locks once again.
      
      Other changes:
      - Take MDL lock before testing if table exists as part of
        CREATE TABLE ... IF EXISTS.  This change was an effect of changes in
        lock_table_name and removes an inconsistency where one could get
        different error messages from CREATE TABLE .. IF EXISTS depending on
        active mdl locks.
        One effect of this change is that we don't binary log CREATE TABLE IF
        EXISTS if the table exists.  This was done because old code was sometimes
        behaving inconsistenly (it was logged some time and not other times)
        and sending the query to the slave could make the slave even more
        inconsistent as there is not guarantee that the new table will have
        the same definition as the old table on the master.
      8cf7e345
    • Vladislav Vaintroub's avatar
    • Monty's avatar
      Added syntax and implementation for BACKUP STAGE's · c53aab97
      Monty authored
      Part of MDEV-5336 Implement LOCK FOR BACKUP
      
      - Changed check of Global_only_lock to also include BACKUP lock.
      - We store latest MDL_BACKUP_DDL lock in thd->mdl_backup_ticket to be able
        to downgrade lock during copy_data_between_tables()
      c53aab97
    • Monty's avatar
      Added new MDL_BACKUP locks for all backup stages · 965311ee
      Monty authored
      Part of MDEV-5336 Implement LOCK FOR BACKUP
      
      - Added new locks to MDL_BACKUP for all stages of backup locks and
        a new MDL lock needed for backup stages.
      - Renamed MDL_BACKUP_STMT to MDL_BACKUP_DDL
      - flush_tables() takes a new parameter that decides what should be flushed.
      - InnoDB, Aria (transactional tables with checksums), Blackhole, Federated
        and Federatedx tables are marked to be safe for online backup. We are
        using MDL_BACKUP_TRANS_DML instead of MDL_BACKUP_DML locks for these
        which allows any DML's to proceed for these tables during the whole
        backup process until BACKUP STAGE COMMIT which will block the final
        commit.
      965311ee
    • Monty's avatar
      Honor lock_wait_timeout when updating stats tables · f386b70b
      Monty authored
      - Call delete_statistics_tables() after lock_table_names in drop tables.
        This avoids a deadlock issue with FTWRL and future backup locks.
      - Added some missing clear_error()
      - Ensure we don't clear error caused by the caller
      - Updated function comments
      f386b70b
    • Monty's avatar
      Simplify test for updateable TABLE_CATEGORY's · e54643a0
      Monty authored
      - Re-numbered enum_table_category to make some tests easier
      - Moved TABLE_CATEGORY_INFORMATION to be first CATEGORY of virtual tables
      - Don't take MDL locks for not updateable table category's
      e54643a0
    • Sergey Vojtovich's avatar
      Acquire global read lock (MDL_BACKUP_STMT) after share is acquired · f1867505
      Sergey Vojtovich authored
      Part of MDEV-5336 Implement LOCK FOR BACKUP
      
      FLUSH TABLE table_names have changed slighty as we are now opening
      tables before taking the MDL lock. The difference is that FLUSH TABLE
      table_name will now be blocked by a table that is waiting for FTWRL.
      There should not be any new deadlocks as part of this change.
      
      The end result is still better in most cases as FTWRL is now only
      waiting for write statements to end, not for read only statements and
      it's not flushing tables in use from the table cache.
      
      Share will be needed to be able to determine if table supports online
      backup. Appropriate metadata lock type in BACKUP namespace will be
      acquired basing on this information.
      
      Also made pending global read lock request to be preferred victim of MDL
      deadlock detector. This allows us to hide some non-fatal deadlocks and
      make FTWRL less likely to break concurrent queries.
      f1867505
    • Sergey Vojtovich's avatar
      Combine GLOBAL and COMMIT namespaces into BACKUP namespace. · 7a9dfdd8
      Sergey Vojtovich authored
      Part of MDEV-5336 Implement LOCK FOR BACKUP
      
      Other things:
      - Added printing of MDL locks to DBUG.
      7a9dfdd8
    • Monty's avatar
      Changed FLUSH TABLES to not change share version · 7fb9d649
      Monty authored
      Part of MDEV-5336 Implement LOCK FOR BACKUP
      
      Originally both table metadata lock and global read lock protection
      were acquired before getting TABLE from table cache. This will be
      reordered in a future commit with MDL_BACKUP_XXX locks so that we
      first take table metadata lock, then get TABLE from table cache, then
      acquire analogue of global read lock.
      
      This patch both simplifies FLUSH TABLES code, makes FLUSH TABLES to
      lock less and also enables FLUSH TABLES code to be used with backup
      locks.
      
      The usage of FLUSH TABLES changes slightly:
      - FLUSH TABLES without any arguments will now only close not used tables
        and tables locked by the FLUSH TABLES connection.  All not used table
        shares will be closed.
        Tables locked by the FLUSH TABLES connection will be reopened and
        re-locked after all others has stoped using the table (as before).
        If there was no locked tables, then FLUSH TABLES is instant and will
        not cause any waits.
        FLUSH TABLES will not wait for any in use table.
      - FLUSH TABLES with a table list, will ensure that the tables are closed
        before statement returns. The code is now only using MDL locks and not
        table share versions, which simplices the code greatly. One visible
        change is that the server will wait for the end of the transaction that
        are using the tables. Before FLUSH TABLES only waited for the statements
        to end.
      Signed-off-by: default avatarMonty <monty@mariadb.org>
      7fb9d649
    • Monty's avatar
      Fixed race condition between flush tables and insert delayed · 7bb3a522
      Monty authored
      FLUSH TABLES kills all insert delayed treads. If this happens at same time
      as a newly started insert delayed query, the insert may fail with either
      and error ("Query interrupted") or the row may be lost.
      This patch fixes this by changing a failed attempt of get_delay_table() to
      convert the query to use a normal insert.
      
      The test case for this patch can be found in the FLUSH TABLES commit after
      this one.
      7bb3a522
    • Monty's avatar
      Optimize flush tables with read lock (FTWRL) to not wait for select's · 163b34fe
      Monty authored
      Part of MDEV-5336 Implement LOCK FOR BACKUP
      
      The idea is that instead of waiting in close_cached_tables() for all
      tables to be closed, we instead call flush_tables() that does:
      - Flush not used objects in table cache to free memory
      - Collect all tables that are open
      - Call HA_EXTRA_FLUSH on the objects, to get them into "closed state"
      - Added HA_EXTRA_FLUSH support to archive and CSV
      - Added multi-user protection to HA_EXTRA_FLUSH in MyISAM and Aria
      
      The benefit compared to old code is:
      - FTWRL doesn't have to wait for long running read operations or
        open HANDLER's
      163b34fe
    • Monty's avatar
      Added API for copying aria tables in mariabackup · 306b7a22
      Monty authored
      306b7a22
    • Monty's avatar
      2aff2f2f
    • Monty's avatar
      Added backup handler calls · 1077f320
      Monty authored
      Part of MDEV-5336 Implement LOCK FOR BACKUP
      
      Added support for backup calls in Aria to protect
      removal of redo logs.
      1077f320
    • Monty's avatar
      Fixed that CREATE TABLE code used the right create_info · ecdf9792
      Monty authored
      - The old code used the original create_info from lex, not the new one
        that includes more information (like OPT_OR_REPLACE).
        The bug was not discovered as the code in lock_table_named() only
        checked for OPT_OR_REPLACE in case of timeout errors.
        As lock_table_names will be fixed as part of BACKUP STAGE's, there
        is no changes in lock_table_names() in this commit.
      - Removed also the 'temporary' copy of statement flags to thd for
        lock_table_names()
      ecdf9792
    • Monty's avatar
      Added timing of bootstrap to mtr · 4f541c5f
      Monty authored
      4f541c5f
    • Monty's avatar
      Cleaned up some MTR tests · 5280af2b
      Monty authored
      - Moved tests depending on utf8 characters from create to create_utf8
      - Fixed some tests in create and sp that wrongly tried to create table
        from non existing table on existing table.  In a later patch we may
        first check if table exists, in which case the error message would
        change.
      - Updated results for partition_debug_tokudb
      5280af2b
    • Monty's avatar
      Fixed test case bug in purge_thread_shutdown · dc91330d
      Monty authored
      dc91330d
    • Monty's avatar
      8561a0ac
    • Monty's avatar
      Fixed BUILD scripts to by default work with galera and tokudb · 84c0563a
      Monty authored
      Bascially this means that all builds with BUILD scripts are done with
      --with-jemalloc=NO      # Required by tokudb
      --with-ssl              # Required on OpenSuse 10.5 to get galera to work
      84c0563a