1. 04 Aug, 2020 4 commits
    • Marko Mäkelä's avatar
      Merge 10.5 into 10.6 · 9a7948e3
      Marko Mäkelä authored
      9a7948e3
    • Marko Mäkelä's avatar
      MDEV-23379 Deprecate&ignore InnoDB concurrency throttling parameters · bbd70fcc
      Marko Mäkelä authored
      The parameters innodb_thread_concurrency and innodb_commit_concurrency
      were useful years ago when both computing resources and the implementation
      of some shared data structures were limited. MySQL 5.0 or 5.1 had trouble
      scaling beyond 8 concurrent connections. Most of the scalability bottlenecks
      have been removed since then, and the transactions per second delivered
      by MariaDB Server 10.5 should not dramatically drop upon exceeding the
      'optimal' number of connections.
      
      Hence, enabling any concurrency throttling for InnoDB actually makes
      things worse. We have seen many customers mistakenly setting this to a
      small value like 16 or 64 and then complaining the server was slow.
      
      Ignoring the parameters allows us to remove some normally unused code
      and data structures, which could slightly improve performance.
      
      innodb_thread_concurrency, innodb_commit_concurrency,
      innodb_replication_delay, innodb_concurrency_tickets,
      innodb_thread_sleep_delay, innodb_adaptive_max_sleep_delay:
      Deprecate and ignore; hard-wire to 0.
      
      The column INFORMATION_SCHEMA.INNODB_TRX.trx_concurrency_tickets
      will always report 0.
      bbd70fcc
    • Daniel Black's avatar
      MDEV-23362: s3 postfix libz -> z · 7438fc4f
      Daniel Black authored
      7438fc4f
    • Daniel Black's avatar
      MDEV-23362: s3 - link to zlib · fd1d6969
      Daniel Black authored
      The function uncompress from the zlib libraries is used
      in s3_get_object in s3_func.c.
      fd1d6969
  2. 02 Aug, 2020 1 commit
    • Marko Mäkelä's avatar
      MDEV-23369 False sharing in page_hash_latch::read_lock_wait() · c12d24e2
      Marko Mäkelä authored
      MDEV-22871 refactored the InnoDB buf_pool.page_hash to use a simple
      rw-lock implementation that avoids a spinloop between non-contended
      read-lock requests, simply using std::atomic::fetch_add() for the
      lock acquisition.
      
      Alas, in a write-heavy stress test on a 56-core system with 1,000
      concurrent client connections, the server would stop processing
      any transactions every now and then. The reason turned out to be
      false sharing. Attaching a debugger to the server during one such
      hang revealed that 22 of the 1,033 threads were polling in
      page_hash_latch::read_lock_wait() on the same object, which appeared
      to be in unlocked state (no readers or writers). All 22 requests were
      for accessing an undo log page, with a distinct page number.
      
      To eliminate such false sharing, we will make buf_pool.page_hash.array
      contain one page_hash_latch per CPU data cache line. On AMD64, this
      will pad the size of the array by 8/7, or almost 15%. For a 50GiB
      buffer pool of 16KiB pages, the buf_pool.page_hash.array would
      grow from 25MiB to 28.6MiB. On other instruction set architectures,
      the incurred memory overhead may be smaller.
      
      Thanks to Vladislav Vaintroub for noticing this anomaly.
      c12d24e2
  3. 01 Aug, 2020 7 commits
  4. 31 Jul, 2020 19 commits
    • Sergei Golubchik's avatar
    • Marko Mäkelä's avatar
      Merge 10.3 into 10.4 · 4db4b773
      Marko Mäkelä authored
      4db4b773
    • Marko Mäkelä's avatar
      Merge 10.3 into 10.4 · 70d4500c
      Marko Mäkelä authored
      70d4500c
    • Marko Mäkelä's avatar
      Merge 10.3 into 10.4 · 9216114c
      Marko Mäkelä authored
      9216114c
    • Alexander Barkov's avatar
      MDEV-23351 Rounding functions return wrong data types for DATE input · dc513dff
      Alexander Barkov authored
      Fixing ROUND(date,0), TRUNCATE(date,x), FLOOR(date), CEILING(date)
      to return the `int(8) unsigned` data type.
      
      Details:
      1. Cleanup: moving virtual implementations
         - Type_handler_temporal_result::Item_func_int_val_fix_length_and_dec()
         - Type_handler_temporal_result::Item_func_round_fix_length_and_dec()
         to Type_handler_date_common. Other temporal data type handlers
         override these methods anyway. So they were only DATE specific.
         This change makes the code clearer.
      2. Backporting DTCollation_numeric from 10.5, to reuse the code easier.
      3. Adding the `preferred_attrs` argument to Item_func_round::fix_arg_int(). Now
         Type_handler_xxx::Item_func_round_val_fix_length_and_dec() work as follows:
         - The INT-alike and YEAR handlers copy preferred_attrs from args[0].
         - The DATE handler passes explicit attributes, to get `int(8) unsigned`.
         - The hex hybrid handler passes NULL, so fix_arg_int() calculates attributes.
      4. Type_handler_date_common::Item_func_int_val_fix_length_and_dec()
         now sets the type handler and attributes to get `int(8) unsigned`.
      dc513dff
    • Oleksandr Byelkin's avatar
      MDEV-21201 No records produced in information_schema query, depending on projection · a8458a23
      Oleksandr Byelkin authored
      In case of NATURAL JOIN / USING mark all field (one table can not be opened
      in any case so optimisation does not worth it).
      
      IMHO table should be checked for used fields and filled after prepare,
      when we will fave whole info about used fields but it is too big change
      for a bugfix. Which will be made later by Serg patch
      a8458a23
    • Marko Mäkelä's avatar
      Merge 10.2 into 10.3 · 78c2a5ab
      Marko Mäkelä authored
      78c2a5ab
    • Thirunarayanan Balathandayuthapani's avatar
      MDEV-14711 Fix-up · 5ec40fbb
      Thirunarayanan Balathandayuthapani authored
      5ec40fbb
    • Marko Mäkelä's avatar
      Merge 10.2 into 10.3 · 8bb2170d
      Marko Mäkelä authored
      8bb2170d
    • Marko Mäkelä's avatar
      Merge 10.2 into 10.3 · 66ec3a77
      Marko Mäkelä authored
      66ec3a77
    • Thirunarayanan Balathandayuthapani's avatar
      MDEV-22511 innodb.truncate_foreign failed in buildbot with wrong error code · a6066e23
      Thirunarayanan Balathandayuthapani authored
      - Adding lock_wait_timeout value as 1 make sure that truncate table
      fails instead of making MDL timeout.
      a6066e23
    • Sergei Golubchik's avatar
    • Marko Mäkelä's avatar
      MDEV-11799 Doublewrite recovery can corrupt data pages · 879ba197
      Marko Mäkelä authored
      The purpose of the InnoDB doublewrite buffer is to make InnoDB
      tolerant against cases where the server was killed in the middle
      of a page write. (In Linux, killing a process may interrupt a
      write system call, typically on a 4096-byte boundary.)
      
      There may exist multiple copies of a page number in the doublewrite
      buffer. Recovery should choose the latest valid copy of the page.
      By design, the FIL_PAGE_LSN must not precede the latest checkpoint LSN
      nor be later than the end of the recovered log.
      
      For page_compressed and encrypted pages, we were missing proper
      consistency checks. In the 10.4 data set generated for in MDEV-23231,
      the data file contained a valid page_compressed page, and an
      identical copy of that page was also present in the doublewrite
      buffer. But, recovery would incorrectly consider the page invalid
      and restore an uncompressed copy of the same page that had been
      written before the log checkpoint. (In fact, no redo log was to
      be applied to that page.)
      
      buf_dblwr_process(): Validate the FIL_PAGE_LSN in the doublewrite
      buffer pages, and always skip page 0, because those pages should
      have been recovered by Datafile::restore_from_doublewrite() if
      necessary.
      
      Datafile::restore_from_doublewrite(): Choose the latest applicable
      page from the doublewrite buffer.
      
      recv_dblwr_t::find_page(): Also validate encrypted or
      page_compressed pages.
      
      recv_dblwr_t::validate_page(): New function to validate a page,
      either a copy in a data file or in the doublewrite buffer.
      Also validate encrypted or page_compressed pages.
      
      This is joint work with Thirunarayanan Balathandayuthapani.
      879ba197
    • Marko Mäkelä's avatar
      MDEV-23198 Crash in REPLACE · f35d1721
      Marko Mäkelä authored
      row_vers_impl_x_locked_low(): clust_offsets may point to memory
      that is allocated by mem_heap_alloc() and may have been freed.
      For initializing clust_offsets, try to use the stack-allocated
      buffer instead of a pointer that may point to freed memory.
      
      This fixes a regression that was introduced in
      commit f0aa073f (MDEV-20950).
      f35d1721
    • Nikita Malyavin's avatar
      MDEV-18042 Server crashes upon adding a non-null date column under... · fd0abc89
      Nikita Malyavin authored
      MDEV-18042 Server crashes upon adding a non-null date column under NO_ZERO_DATE with ALGORITHM=INPLACE
      
      accept table_name and db_name instead of table_share in make_truncated_value_warning
      fd0abc89
    • Nikita Malyavin's avatar
      MDEV-19338 InnoDB: Failing assertion: !cursor->index->is_committed() · 91ebf184
      Nikita Malyavin authored
      Call mark_columns_per_binlog_row_image before find_row() to set up table->vcol_set early,
      so the virtual column value will be updated after record read (ha_rnd_pos/ha_index_next/etc)
      by table->update_virtual_fields() call
      91ebf184
    • Sergei Golubchik's avatar
      MDEV-23237 mariadb.sys has too many privileges · 153cd6a2
      Sergei Golubchik authored
      don't GRANT UPDATE ON mysql.global_priv TO mariadb.sys@localhost;
      153cd6a2
    • Sergei Golubchik's avatar
      MDEV-23009 SIGSEGV in get_field from acl_load (on optimized builds) · 78f09b12
      Sergei Golubchik authored
      Classes that handle privilege tables (like Tables_priv_table)
      could read some columns conditionally but they expect a certain
      minimal number of colunms always to exist.
      
      Add a check for a minimal required number of columns in privilege tables,
      don't use a table that has fewer columns than required.
      78f09b12
    • Alexander Barkov's avatar
      MDEV-23337 Rounding functions create a wrong data type for integer input · a874b6c4
      Alexander Barkov authored
      1. Fixing ROUND(x) and TRUNCATE(x,0) with TINYINT, SMALLINT, MEDIUMINT, BIGINT
         input to preserve the exact data type of the argument when it's possible.
      
      2. Fixing FLOOR(x) and CEILING(x) with TINYINT, SMALLINT, MEDIUMINT, BIGINT
        to preserve the exact data type of the argument.
      
      3. Adding dedicated Type_handler_year::Item_func_round_fix_length_and_dec()
        to easier handle ROUND(x) and TRUNCATE(x,y) for the YEAR(2) and YEAR(4)
        input. They still return INT(2) UNSIGNED and INT(4) UNSIGNED correspondingly,
        as before.
      a874b6c4
  5. 30 Jul, 2020 9 commits
    • Marko Mäkelä's avatar
      MDEV-23334 Crash in rec_get_nth_cfield()/rec_offs_validate() · 6053eb1c
      Marko Mäkelä authored
      rec_get_nth_cfield(): Remove a bogus debug assertion.
      The function may be invoked by innobase_rec_to_mysql()
      for reporting a duplicate key error during CREATE UNIQUE INDEX
      or ALTER TABLE...ADD UNIQUE KEY, and in that case the record
      will be missing the 5-byte or 6-byte fixed header.
      
      It turns out that in every other code path leading to
      rec_get_nth_cfield() we either invoked rec_get_offsets()
      ourselves or asserted rec_offs_validate(). So, we can
      safely remove the assertion and make debug builds
      smaller and faster.
      6053eb1c
    • Krunal Bauskar's avatar
      MDEV-23030: ARM crash on Raspberry Pi 4 · c69520c9
      Krunal Bauskar authored
      MariaDB adopted a hardware optimized crc32c approach on ARM64 starting 10.5.
      Said implementation of crc32c needs support from target hardware for crc32
      and pmull instructions. Existing logic is checking only for crc32 support
      from target hardware through a runtime check and so if target hardware
      doesn't support pmull it would cause things to fail/crash.
      
      Expanded runtime check to ensure pmull support is also checked on the target
      hardware along with existing crc32.
      
      Thanks to Marko and Daniel for review.
      c69520c9
    • Jan Lindström's avatar
      Enable test for testing. · c3958ae4
      Jan Lindström authored
      c3958ae4
    • Vladislav Vaintroub's avatar
      MDEV-21101 skip test for embedded · 0435fcf9
      Vladislav Vaintroub authored
      0435fcf9
    • Thirunarayanan Balathandayuthapani's avatar
      MDEV-23332 Index online status assert failure in btr_search_drop_page_hash_index · 8a612314
      Thirunarayanan Balathandayuthapani authored
      Problem:
      ========
      In row_merge_drop_indexes(), InnoDB drops only the index from
      dictionary and frees the index pages but it maintains the index
      object if the table is being used by other DML threads. It sets
      the online status of the index to ONLINE_INDEX_ABORTED_DROPPED.
      Removing the index from dictionary doesn't remove the
      corressponding ahi entries of the index. When block is being
      reused, InnoDB tries to remove ahi entries for the block and
      it fails if index online status is ONLINE_INDEX_ABORTED_DROPPED.
      
      Fix:
      ====
      MDEV-22456 allows the index ahi entries to be dropped lazily.
      so checking online status in btr_search_drop_page_hash_index()
      is meaningless and should be removed.
      8a612314
    • Vladislav Vaintroub's avatar
      MDEV-21101 unexpected wait_timeout with pool-of-threads · 71015d84
      Vladislav Vaintroub authored
      Due to restricted size of the threadpool, execution of client queries can
      be delayed (queued) for a while. This delay was interpreted as client
      inactivity, and connection is closed, if client idle time + queue time
      exceeds wait_timeout.
      
      But users did not expect queue time to be included into wait_timeout.
      
      This patch changes the behavior. We don't close connection anymore,
      if there is some unread data present on connection,
      even if wait_timeout is exceeded. Unread data means that client
      was not idle, it sent a query, which we did not have time to process yet.
      71015d84
    • Marko Mäkelä's avatar
      MDEV-23339 innodb_force_recovery=2 may still abort the rollback of recovered transactions · c5d4dd25
      Marko Mäkelä authored
      trx_rollback_active(), trx_rollback_resurrected(): Replace
      an incorrect condition that we failed to replace in
      commit b68f1d84 (MDEV-21217).
      c5d4dd25
    • Alexander Barkov's avatar
      MDEV-23323 Rounding functions return a wrong data type for a BIT, ENUM, SET argument · 6d3186e3
      Alexander Barkov authored
      Implementing dedicated fixing methods:
      - Type_handler_bit::Item_func_round_fix_length_and_dec()
      - Type_handler_bit::Item_func_int_val_fix_length_and_dec()
      - Type_handler_typelib::Item_func_round_fix_length_and_dec()
      
      because the inherited methods did not work well.
      
      Fixing:
      - Type_handler_typelib::Item_func_int_val_fix_length_and_dec
        It did not work well, because it used args[0]->max_length to
        calculate the result data type. In case of ENUM and SET it was
        not correct, because in FLOOR() and CEILING() context
        ENUM and SET return not more than 5 digits (65535 is the biggest
        possible value).
      
      Misc:
      - Changing the API of
          Type_handler_bit::Bit_decimal_notation_int_digits(const Item *item)
        to a more generic form:
          Type_handler_bit::Bit_decimal_notation_int_digits_by_nbits(uint nbits)
      
      - Fixing Type_handler_bit::Bit_decimal_notation_int_digits_by_nbits() to
        return the exact number of decimal digits for all nbits 1..64.
        The old implementation was approximate.
        This change gives better (more precise) data types.
      6d3186e3
    • Otto Kekäläinen's avatar
      Deb: Make RocksDB plugin depend on python3 as myrocks_hotbackup needs it · f99de891
      Otto Kekäläinen authored
      This fixes the Lintian error and thus the Lintian override can also be
      removed.
      f99de891