1. 13 Dec, 2019 1 commit
  2. 12 Dec, 2019 1 commit
  3. 11 Dec, 2019 1 commit
    • Eugene Kosov's avatar
      MDEV-16678 Prefer MDL to dict_sys.latch for innodb background tasks · f4b42846
      Eugene Kosov authored
      Use std::queue backed by std::deque instead of list because it does less
      allocations which still having O(1) push and pop operations.
      
      Also store trx_purge_rec_t directly, because its only 16 bytes and allocating
      it is to wasteful. This should be faster.
      
      purge_node_t::purge_node_t: container is already empty after creation
      
      purge_node_t::end(): replace clearing container with assertion that it's clear
      f4b42846
  4. 10 Dec, 2019 2 commits
    • Marko Mäkelä's avatar
      MDEV-16678: Fix a problem with duplicate #sql2 table names · adb117cf
      Marko Mäkelä authored
      row_drop_table_for_mysql(): If a #sql2 table is open in another
      thread (purge) while we attempting to drop it, rename it to #sql-ib
      name to hide it from the SQL layer.
      
      Adjust tests accordingly to hide #sql-ib tables, which can
      continue to exist until the background DROP TABLE completes.
      adb117cf
    • Marko Mäkelä's avatar
      MDEV-16678 Prefer MDL to dict_sys.latch for innodb background tasks · ea37b144
      Marko Mäkelä authored
      This is joint work with Thirunarayanan Balathandayuthapani.
      The MDL interface between InnoDB and the rest of the server
      (in storage/innobase/dict/dict0dict.cc and in include/)
      is my work, while most everything else is Thiru's.
      
      The collection of InnoDB persistent statistics and the
      defragmentation were not refactored to use MDL. They will
      keep relying on lower-level interlocking with
      fil_check_pending_operations().
      
      The purge of transaction history and the background operations on
      fulltext indexes will use MDL. We will revert
      commit 2c4844c9
      (MDEV-17813) because thanks to MDL, purge cannot conflict
      with DDL operations anymore. For a similar reason, we will remove
      the MDEV-16222 test case from gcol.innodb_virtual_debug_purge.
      
      Purge is essentially replacing all use of the global dict_sys.latch
      with MDL. Purge will skip the undo log records for tables whose names
      start with #sql-ib or #sql2. Theoretically, such tables might
      be renamed back to visible table names if TRUNCATE fails to
      create a new table, or the final rename in ALTER TABLE...ALGORITHM=COPY
      fails. In that case, purge could permanently leave some garbage
      in the table. Such garbage will be tolerated; the table would not
      be considered corrupted.
      
      To avoid repeated MDL releases and acquisitions,
      trx_purge_attach_undo_recs() will sort undo log records by table_id,
      and purge_node_t will keep the MDL and table handle open for multiple
      successive undo log records.
      
      get_purge_table(): A new accessor, used during the purge of
      history for indexed virtual columns. This interface should ideally
      not exist at all.
      
      thd_mdl_context(): Accessor of THD::mdl_context.
      Wrapped in a new thd_mdl_service.
      
      dict_get_db_name_len(): Define inline.
      
      dict_acquire_mdl_shared(): Acquire explicit shared MDL on a table name
      if needed.
      
      dict_table_open_on_id(): Return MDL_ticket, if requested.
      
      dict_table_close(): Release MDL ticket, if requested.
      
      dict_fts_index_syncing(), dict_index_t::index_fts_syncing: Remove.
      row_drop_table_for_mysql() no longer needs to check these, because
      MDL guarantees that a fulltext index sync will not be in progress
      while MDL_EXCLUSIVE is protecting a DDL operation.
      
      dict_table_t::parse_name(): Parse the table name for acquiring MDL.
      
      purge_node_t::undo_recs: Change the type to std::list<trx_purge_rec_t*>
      (different container, and storing also roll_ptr).
      
      purge_node_t: Add mdl_ticket, last_table_id, purge_thd, mdl_hold_recs
      for acquiring MDL and for keeping the table open across multiple
      undo log records.
      
      purge_vcol_info_t, row_purge_store_vsec_cur(), row_purge_restore_vsec_cur():
      Remove. We will acquire the MDL earlier.
      
      purge_sys_t::heap: Added, for reading undo log records.
      
      fts_sync_during_ddl(): Invoked during ALGORITHM=INPLACE operations
      to ensure that fts_sync_table() will not conflict with MDL_EXCLUSIVE.
      Uses fts_t::sync_message for bookkeeping.
      ea37b144
  5. 09 Dec, 2019 3 commits
    • Vladislav Vaintroub's avatar
      MDEV-21262 MTR does not work with Windows ASAN builds · e47bd007
      Vladislav Vaintroub authored
      Do not use suppressions on Windows, the ASAN_OPTIONS can't be parsed
      whenever full file names are used, because file names contain ':'
      which is ASAN_OPTIONS delimeter.
      e47bd007
    • Vladislav Vaintroub's avatar
      MDEV-16264 - some improvements · 66de4fef
      Vladislav Vaintroub authored
      - wait notification, tpool_wait_begin/tpool_wait_end - to notify the
      threadpool that current thread is going to wait
      
      Use it to wait for IOs to complete and also when purge waits for workers.
      66de4fef
    • Marko Mäkelä's avatar
      MDEV-21259 Assertion failed in mtr_t::write() · d3b2625b
      Marko Mäkelä authored
      btr_free_externally_stored_field(): Pass w=mtr_t::OPT to
      note that the BTR_EXTERN_LEN is not necessarily changing
      when a multi-page ROW_FORMAT=COMPRESSED off-page column
      is being freed, and to allow redundant writes to the redo
      log to be optimized away.
      
      Ever since commit 56f6dab1
      the refactored function mtr_t::write() asserts by default
      that the page contents is being changed.
      d3b2625b
  6. 07 Dec, 2019 1 commit
  7. 06 Dec, 2019 2 commits
  8. 05 Dec, 2019 5 commits
    • Kentoku's avatar
      fix compiler warnings · 1b040ce5
      Kentoku authored
      1b040ce5
    • Marko Mäkelä's avatar
      MDEV-21174: Fix undefined behaviour · 6189774c
      Marko Mäkelä authored
      ibuf_bitmap_page_set_bits(): Do not attempt to shift by a negative amount.
      This bug was introduced in commit 87839258.
      6189774c
    • Marko Mäkelä's avatar
      MDEV-21225: Fix warnings · 46fc3bdb
      Marko Mäkelä authored
      Datafile::find_space_id(): Remove an unused variable
      
      buf_chunk_init(): warning C4319:  '~': zero extending 'ulong' to 'ulint'
      of greater size
      46fc3bdb
    • Alexander Barkov's avatar
      A cleanup for MDEV-17088 Provide tools to encode/decode mysql-encoded file system names · 2c7b6214
      Alexander Barkov authored
      - Load and convert the entire input file at once,
        rather than reading string-by-string using fgets().
        This change makes it possible to convert from UCS2, UTF16, UTF32 data.
      - Adding the --delimiter command, to treat the specified
        characters as delimiters rather than data to convert.
        Useful in combination with `-f filename` or `-t filename`.
        The delimiter characters are not converted,
        they are copied from the input to the output as is.
      - Adding diagnostics with line number and position if:
        * an illegal input byte sequence was found
        * a character cannot be converted to the target character set
      2c7b6214
    • Marko Mäkelä's avatar
      MDEV-21225 Remove ut_align() and use aligned_malloc() · 42a4ae54
      Marko Mäkelä authored
      Before commit 90c52e52 introduced
      aligned_malloc(), InnoDB always used a pattern of over-allocating
      memory and invoking ut_align() to guarantee the desired alignment.
      
      It is cleaner to invoke aligned_malloc() and aligned_free() directly.
      
      ut_align(): Remove. In assertions, ut_align_down() can be used instead.
      42a4ae54
  9. 04 Dec, 2019 8 commits
    • Marko Mäkelä's avatar
      MDEV-21216: Remove fsp_header_get_space_id() · 504202bd
      Marko Mäkelä authored
      The function fsp_header_get_space_id() returns ulint instead of
      uint32_t, only to be able to complain that the two adjacent
      tablespace ID fields in the page differ. Remove the function,
      and merge the check to the callers.
      
      Also, make some more use of aligned_malloc().
      504202bd
    • Marko Mäkelä's avatar
      MDEV-21174: Fix Galera · 2352f516
      Marko Mäkelä authored
      trx_rseg_write_wsrep_checkpoint(): Use mtr_t::OPT, because
      much of the time, the redo log writes would be redundant.
      This was broken in commit 56f6dab1.
      2352f516
    • Marko Mäkelä's avatar
      MDEV-19514: Update stale comments · 6b5cdd4f
      Marko Mäkelä authored
      6b5cdd4f
    • Marko Mäkelä's avatar
      MDEV-21216 InnoDB does dirty read of TRX_SYS page before recovery · 95e90326
      Marko Mäkelä authored
      InnoDB startup was discovering undo tablespaces in a dirty way.
      It was reading a possibly stale copy of the TRX_SYS page before
      processing any redo log records.
      
      srv_start(): Do not call buf_pool_invalidate(). Invoke
      trx_rseg_get_n_undo_tablespaces() after the recovery has been initiated.
      
      recv_recovery_from_checkpoint_start(): Assert that the buffer pool is
      empty. This used to be guaranteed by the buf_pool_invalidate() call.
      
      trx_rseg_get_n_undo_tablespaces(): Move to the calling compilation unit,
      and reimplement in a simpler way.
      
      srv_undo_tablespace_create(): Remove the constant parameter
      size=SRV_UNDO_TABLESPACE_SIZE_IN_PAGES.
      
      srv_undo_tablespace_open(): Reimplement in a cleaner way, with
      more robust error handling.
      
      srv_all_undo_tablespaces_open(): Split from srv_undo_tablespaces_init().
      
      srv_undo_tablespaces_init(): Read all "undo001","undo002" tablespace
      files directly, without consulting the TRX_SYS page via calling
      trx_rseg_get_n_undo_tablespaces().
      
      This is joint work with Thirunarayanan Balathandayuthapani.
      95e90326
    • Marko Mäkelä's avatar
    • Vladislav Vaintroub's avatar
      Part2: MDEV-12518 Unify sql_yacc.yy and sql_yacc_ora.yy · bf303419
      Vladislav Vaintroub authored
      Generate sql_yacc_ora.yy from sql_yacc.yy , by using cmake script
      which does some simple string(REPLACE)
      bf303419
    • Alexander Barkov's avatar
      Part1: MDEV-12518 Unify sql_yacc.yy and sql_yacc_ora.yy · 3f9d7072
      Alexander Barkov authored
      Mixing %type and %expect declarations:
      - sql_mode=ORACLE declarations look like an empty C code section
        inside sql_yacc.yy, consisting of an inactive #ifdef..#endif block.
      - sql_mode=DEFAULT declarations look like an empty C code section
        inside sql_yacc_ora.yy, consisting of an inactive #ifdef..#endif block.
      
      Mixing rules:
      - Adding a special rule _empty to the shared rule section.
      - Changing all instances of /*Empty*/ in sql_mode=DEFAULT and sql_mode=ORACLE
        specific sections to _empty.
      - Changing the rest of C style comments /*xxx*/ in
        sql_mode=DEFAULT and sql_mode=ORACLE specific blocks to C++ style: //xxx
      - Mixing sql_yacc.yy and sql_yacc_ora.yy, so
        sql_mode=ORACLE specific blocks sit in a comment inside sql_yacc.yy, and
        sql_mode=DEFAULT specific blocks sit in a comment inside sql_yacc_ora.yy.
      3f9d7072
    • Marko Mäkelä's avatar
      MDEV-21205 Assertion failure in btr_sec_min_rec_mark · bb459416
      Marko Mäkelä authored
      In commit af5947f4
      the function btr_discard_page() is invoking btr_set_min_rec_mark()
      with the wrong buf_block_t* object. node_ptr is on merge_block,
      not block.
      
      btr_discard_page(): Remove the variables merge_page, page, and
      always refer to block->frame or merge_block->frame instead.
      Also, limit the scope of node_ptr and avoid duplicated conditions.
      
      btr_set_min_rec_mark(): Add a template parameter, so that the
      caller can specify whether the page is supposed to have a left sibling.
      Otherwise, the assertion (which was introduced in the same commit)
      would fail in btr_discard_page().
      bb459416
  10. 03 Dec, 2019 13 commits
    • Daniel Bartholomew's avatar
      bump the VERSION · 3b9a978a
      Daniel Bartholomew authored
      3b9a978a
    • Marko Mäkelä's avatar
      MDEV-21174: Fix the 32-bit build · 2ac0e64c
      Marko Mäkelä authored
      mtr_t::write(): Add explicit narrowing type casts to avoid warnings
      about lossy implicit conversions.
      2ac0e64c
    • Marko Mäkelä's avatar
      MDEV-21174: Replace mlog_write_string() with mtr_t::memcpy() · af5947f4
      Marko Mäkelä authored
      mtr_t::memcpy(): Replaces mlog_write_string(), mlog_log_string().
      The buf_block_t is passed a parameter, so that
      mlog_write_initial_log_record_low() can be used instead of
      mlog_write_initial_log_record_fast().
      
      fil_space_crypt_t::write_page0(): Remove the fil_space_t* parameter.
      af5947f4
    • Marko Mäkelä's avatar
      MDEV-21174: Replace mlog_memset() with mtr_t::memset() · 87839258
      Marko Mäkelä authored
      Passing buf_block_t helps us avoid calling
      mlog_write_initial_log_record_fast() and page_get_page_no(),
      and allows us to implement more debug checks, such as
      that on ROW_FORMAT=COMPRESSED index pages, only the page header
      may be modified by MLOG_MEMSET records.
      
      fseg_n_reserved_pages(): Add a buf_block_t parameter.
      87839258
    • Marko Mäkelä's avatar
      Cleanup: Remove some page_get_page_no() calls · caea64df
      Marko Mäkelä authored
      Refer to buf_page_t::id instead of parsing the tablespace identifier
      or page number from the buffer pool page.
      caea64df
    • Marko Mäkelä's avatar
      MDEV-21174: Replace mlog_write_ulint() with mtr_t::write() · 56f6dab1
      Marko Mäkelä authored
      mtr_t::write(): Replaces mlog_write_ulint(), mlog_write_ull().
      Optimize away writes if the page contents does not change,
      except when a dummy write has been explicitly requested.
      
      Because the member function template takes a block descriptor as a
      parameter, it is possible to introduce better consistency checks.
      Due to this, the code for handling file-based lists, undo logs
      and user transactions was refactored to pass around buf_block_t.
      56f6dab1
    • Marko Mäkelä's avatar
      MDEV-21174: Cleanup MLOG_PAGE_CREATE · 504823bc
      Marko Mäkelä authored
      page_create_write_log(), mlog_write_initial_log_record():
      Merge to the only caller, and use
      mlog_write_initial_log_record_low() for writing the log record.
      504823bc
    • Marko Mäkelä's avatar
      MDEV-16264: Minor cleanup · 57444a3b
      Marko Mäkelä authored
      aio_linux::m_max_io_count: Unused data member; remove.
      
      aiocb::m_ret_len: Declare as the more compatible type size_t.
      Unfortunately, ssize_t is not available on Microsoft Visual Studio.
      57444a3b
    • Marko Mäkelä's avatar
      MDEV-12353 preparation: Do not write MLOG_REC_MIN_MARK · cd92c6c8
      Marko Mäkelä authored
      btr_set_min_rec_mark(): Write MLOG_1BYTE instead of
      MLOG_REC_MIN_MARK or MLOG_COMP_REC_MIN_MARK.
      
      On ROW_FORMAT=COMPRESSED pages, the minimum record flag is not stored
      at all. The flag is computed for the uncompressed page by
      page_zip_decompress(). Hence, nothing needs to be logged for
      ROW_FORMAT=COMPRESSED tables for this operation.
      
      To facilitate crash-upgrade and hot backup from older versions,
      we will retain the code to parse and apply the old log record types
      MLOG_REC_MIN_MARK and MLOG_COMP_REC_MIN_MARK.
      cd92c6c8
    • Marko Mäkelä's avatar
      MDEV-12353 preparation: Do not write MLOG_FILE_WRITE_CRYPT_DATA · 8ebd91c1
      Marko Mäkelä authored
      The MLOG_FILE_WRITE_CRYPT_DATA record was completely redundant.
      It can be replaced with a single MLOG_WRITE_STRING record.
      
      To facilitate upgrade from older versions, we will retain
      fil_parse_write_crypt_data().
      
      fil_crypt_parse(): Recover fil_space_crypt_t::write_page0().
      
      fil_space_crypt_t::write_page0(): Write everything in a single
      MLOG_WRITE_STRING for easy parsing.
      
      fil_space_crypt_t::page0_offset: Remove.
      8ebd91c1
    • Marko Mäkelä's avatar
      MDEV-21133: Remove buf_frame_copy() · bf2cc467
      Marko Mäkelä authored
      bf2cc467
    • Aleksey Midenkov's avatar
      MDEV-17554 versioning partition tests reorganize · 6f899468
      Aleksey Midenkov authored
      Moved partition_rotation, partition_innodb to partition.test
      6f899468
    • Aleksey Midenkov's avatar
      MDEV-17554 history partitioning cleanups · 9ed8d364
      Aleksey Midenkov authored
      * Fixed missed warning on condition boundary
      * REORGANIZE cases
      * vers_utils.h removed
      * test cases cleanup
      9ed8d364
  11. 02 Dec, 2019 3 commits