1. 04 Jun, 2021 1 commit
  2. 03 Jun, 2021 1 commit
    • Igor Babaev's avatar
      MDEV-25714 Join using derived with aggregation returns incorrect results · 663bc849
      Igor Babaev authored
      If a join query uses a derived table (view / CTE) with GROUP BY clause then
      the execution plan for such join may employ split optimization. When this
      optimization is employed the derived table is not materialized. Rather only
      some partitions of the derived table are subject to grouping. Split
      optimization can be applied only if:
      - there are some indexes over the tables used in the join specifying the
        derived table whose prefixes partially cover the field items used in the
        GROUP BY list (such indexes are called splitting indexes)
      - the WHERE condition of the join query contains conjunctive equalities
        between columns of the derived table that comprise major parts of
        splitting indexes and columns of the other join tables.
      
      When the optimizer evaluates extending of a partial join by the rows of the
      derived table it always considers a possibility of using split optimization.
      Different splitting indexes can be used depending on the extended partial
      join. At some rare conditions, for example, when there is a non-splitting
      covering index for a table joined in the join specifying the derived table
      usage of a splitting index to produce rows needed for grouping may be still
      less beneficial than usage of such covering index without any splitting
      technique. The function JOIN_TAB::choose_best_splitting() must take this
      into account.
      
      Approved by Oleksandr Byelkin <sanja@mariadb.com>
      663bc849
  3. 02 Jun, 2021 3 commits
  4. 01 Jun, 2021 1 commit
  5. 31 May, 2021 2 commits
    • Julius Goryavsky's avatar
      MDEV-25818: RSYNC SST failed due to busy port · 2fb44078
      Julius Goryavsky authored
      This commit reduces the likelihood of getting a busy port on
      quick restarts with rsync SST (problem MDEV-25818) and fixes
      a number of other flaws in SST scripts, adds new functionality,
      and also synchronizes the xtrabackup-v2 script with the
      mariabackup script (the latter applies only to the 10.2 branch):
      
       1) SST via rsync: rsync and stunnel does not always get the right
          time to complete by correctly handling SIGTERM. These utilities
          are now given more time to complete normally (via normal SIGTERM
          processing) before we move on to using "kill -9";
       2) SST via rsync: attempts to terminate an rsync or stunnel process
          (via "kill" utility) are only made if it did not terminated on
          its own;
       3) SST via rsync: if a combination of stunnel and rsync is used,
          then we need to wait for both utilities to finish or stop, not
          just one of them;
       4) The config file and pid file for stunnel are now deleted after
          successful completion of SST on the donor node;
       5) The configs and pid files from rsync and stunnel should not be
          deleted unless these utilities succeed (or are sucessfully
          terminated) on the joiner node;
       6) The configs and pid files now excluded from transfer via rsync;
       7) Spaces in paths are now valid for config files as well (when
          used with SST via rsync or mariabackup / xtrabackup[-v2]);
       8) SST via mariabackup: added preliminary verification of keys and
          certificates that are used when establishing a connection using
          SSL (to avoid long timeouts and improve diagnostics) - by analogy
          with how it is done for the xtrabackup-v2 (plus check for CA file),
          while that check is skipped if the user does not have openssl
          installed (or does not have diff utility);
       9) Added backup-threads=<n> configuration option which adds
          "--parallel=<n>" for mariabackup / xtrabackup at backup and
          move-back stages;
      10) Added encrypt-threads and encrypt-chunk-size configuration
          options for xbcrypt management (when xbcrypt is used);
      11) Small optimization: checking the socat version and adding
          a file with parameters for 2048-bit Diffie-Hellman (if necessary)
          is done only if the user has not specified "dhparam=" in the
          "sockopt" option value;
      12) SST via rsync now supports "backup-threads" configuration option
          (in server-related sections or in the "[sst]");
      13) Determining the number of available processors is now supported
          for FreeBSD + mariabackup/xtrabackup: before that we might have
          problems with "--compact" (rebuild indexes) or qpress on FreeBSD;
      14) The check_pid() function should not raise an error state in
          the rare cases when the pid file was created, but it is empty,
          or if it is deleted right during the check, or when zero is read
          from the pid file;
      15) Iproved templates that are used to check if a requested socket
          is "listening" when using the ss utility;
      16) Shortened some other templates for socket state utilities;
      17) Temporary files created by mariabackup / xtrabackup are moved
          to a separate subdirectory inside tmpdir (so they don't get
          mixed with other temporary files, which can make debugging
          more difficult);
      18) 10.2 only: the script for SST via xtrabackup-v2 has been brought
          in full compliance with all the bugfixes made for mariabackup (as
          it previously contained many flaws compared to the updated script
          for mariabackup).
      2fb44078
    • Vladislav Vaintroub's avatar
  6. 30 May, 2021 1 commit
    • Dmitry Shulga's avatar
      MDEV-25576: The statement EXPLAIN running as regular statement and as prepared... · 91bde0fb
      Dmitry Shulga authored
      MDEV-25576: The statement EXPLAIN running as regular statement and as prepared statement produces different results for UPDATE with subquery
      
      Both EXPLAIN and EXPLAIN EXTENDED statements produce different results set
      in case it is run in normal way and in PS mode for the statements
      UPDATE/DELETE with subquery.
      
      The use case below reproduces the issue:
      MariaDB [test]> CREATE TABLE t1 (c1 INT KEY) ENGINE=MyISAM;
      Query OK, 0 rows affected (0,128 sec)
      
      MariaDB [test]> CREATE TABLE t2 (c2 INT) ENGINE=MyISAM;
      Query OK, 0 rows affected (0,023 sec)
      
      MariaDB [test]> CREATE TABLE t3 (c3 INT) ENGINE=MyISAM;
      Query OK, 0 rows affected (0,021 sec)
      
      MariaDB [test]> EXPLAIN EXTENDED UPDATE t3 SET c3 =
          -> ( SELECT COUNT(d1.c1) FROM ( SELECT a11.c1 FROM t1 AS a11
          -> STRAIGHT_JOIN t2 AS a21 ON a21.c2 = a11.c1 JOIN t1 AS a12
          -> ON a12.c1 = a11.c1 ) d1 );
      +------+-------------+-------+------+---------------+------+---------+------+------+----------+--------------------------------+
      | id   | select_type | table | type | possible_keys | key  | key_len | ref  | rows | filtered | Extra                          |
      +------+-------------+-------+------+---------------+------+---------+------+------+----------+--------------------------------+
      |    1 | PRIMARY     | t3    | ALL  | NULL          | NULL | NULL    | NULL |    0 |   100.00 |                                |
      |    2 | SUBQUERY    | NULL  | NULL | NULL          | NULL | NULL    | NULL | NULL |     NULL | Impossible WHERE noticed after reading const tables
      +------+-------------+-------+------+---------------+------+---------+------+------+----------+--------------------------------+
      2 rows in set (0,002 sec)
      
      MariaDB [test]> PREPARE stmt FROM
          -> EXPLAIN EXTENDED UPDATE t3 SET c3 =
          -> ( SELECT COUNT(d1.c1) FROM ( SELECT a11.c1 FROM t1 AS a11
          -> STRAIGHT_JOIN t2 AS a21 ON a21.c2 = a11.c1 JOIN t1 AS a12
          -> ON a12.c1 = a11.c1 ) d1 );
      Query OK, 0 rows affected (0,000 sec)
      Statement prepared
      
      MariaDB [test]>  EXECUTE stmt;
      +------+-------------+-------+------+---------------+------+---------+------+------+----------+--------------------------------+
      | id   | select_type | table | type | possible_keys | key  | key_len | ref  | rows | filtered | Extra                          |
      +------+-------------+-------+------+---------------+------+---------+------+------+----------+--------------------------------+
      |    1 | PRIMARY     | t3    | ALL  | NULL          | NULL | NULL    | NULL |    0 |   100.00 |                                |
      |    2 | SUBQUERY    | NULL  | NULL | NULL          | NULL | NULL    | NULL | NULL |     NULL | no matching row in const table |
      +------+-------------+-------+------+---------------+------+---------+------+------+----------+--------------------------------+
      2 rows in set (0,000 sec)
      
      The reason by that different result sets are produced is that on execution
      of the statement 'EXECUTE stmt' the flag SELECT_DESCRIBE not set
      in the data member SELECT_LEX::options for instances of SELECT_LEX that
      correspond to subqueries used in the UPDTAE/DELETE statements.
      
      Initially, these flags were set on parsing the statement
        PREPARE stmt FROM "EXPLAIN EXTENDED UPDATE t3 SET ..."
      but latter they were reset before starting real execution of
      the parsed query during handling the statement 'EXECUTE stmt';
      
      So, to fix the issue the functions mysql_update()/mysql_delete()
      have been modified to set the flag SELECT_DESCRIBE forcibly
      in the data member SELECT_LEX::options for the primary SELECT_LEX
      of the UPDATE/DELETE statement.
      91bde0fb
  7. 27 May, 2021 5 commits
  8. 26 May, 2021 3 commits
  9. 25 May, 2021 2 commits
    • Sergei Golubchik's avatar
      MDEV-25730: maria.repair test fails with valgrind · d8fa71a0
      Sergei Golubchik authored
      cherry-pick commit: 1fff2398
      MDEV-22530 post push fixes from 10.6.
      
      Followup. If the KILL happens - report it as a failure,
      don't eat it up silently. Note that this has to be done after `table_name`
      is populated, so that the error message could show it.
      d8fa71a0
    • Julius Goryavsky's avatar
      MDEV-21192: SST failing when enabling IPV6 · fe7e44d8
      Julius Goryavsky authored
      The following features have been added:
      
      1) Automatic addition of the pf = ip6 option for socat
         when it can be recognized by the format of the connection
         address;
      2) Automatically add or remove extra commas at the beginning
         and at the end of sockopt, for example, sockopt='pf=ip6'
         and sockopt=',pf=ip6' work equally well;
      
      Also, due to interference in the code of the get_transfer()
      function, I also refactored it and now:
      
      3) encrypt = 4 is supported not only for xtrabackup-v2,
         but also for mariabackup - this can help with migration
         from Percona;
      4) Improved setting of 'commonname' option for encrypt=3
         and encrypt=4 modes;
      fe7e44d8
  10. 24 May, 2021 3 commits
    • Julius Goryavsky's avatar
      MDEV-15730: rename --stream=xbstream to --stream=mbstream · 81f94c26
      Julius Goryavsky authored
      mbstream is already supported as a format name after MDEV-24580,
      but additional code refactoring has been done to correctly display
      the format name in log files and to check if the mbstream utility
      is in the path. Also, for xtrabackup-v2 (only available in the 10.2)
      both utilities are supported - both xbstram and mbstream, since they
      are interchangeable in this context. In this case, the original
      innobackupex always receives the correct --stream=xbstream option
      as input, but the user can actually try to use the mbstream utility
      during the transfer (if the user explicitly specifies this in the
      configuration file).
      81f94c26
    • Marko Mäkelä's avatar
      Merge 10.2 into 10.3 · 1864a8ea
      Marko Mäkelä authored
      1864a8ea
    • Thirunarayanan Balathandayuthapani's avatar
      MDEV-25663 Double free of transaction during truncate operation · 5c75ba9c
      Thirunarayanan Balathandayuthapani authored
      - Patch addresses the problem to fix double free of
      transaction if it is own transaction.
      5c75ba9c
  11. 23 May, 2021 2 commits
  12. 22 May, 2021 7 commits
  13. 21 May, 2021 5 commits
    • Igor Babaev's avatar
      MDEV-23886 Reusing CTE inside a function fails with table doesn't exist · 43c9fcef
      Igor Babaev authored
      In the code existed just before this patch binding of a table reference to
      the specification of the corresponding CTE happens in the function
      open_and_process_table(). If the table reference is not the first in the
      query the specification is cloned in the same way as the specification of
      a view is cloned for any reference of the view. This works fine for
      standalone queries, but does not work for stored procedures / functions
      for the following reason.
      When the first call of a stored procedure/ function SP is processed the
      body of SP is parsed. When a query of SP is parsed the info on each
      encountered table reference is put into a TABLE_LIST object linked into
      a global chain associated with the query. When parsing of the query is
      finished the basic info on the table references from this chain except
      table references to derived tables and information schema tables is put
      in one hash table associated with SP. When parsing of the body of SP is
      finished this hash table is used to construct TABLE_LIST objects for all
      table references mentioned in SP and link them into the list of such
      objects passed to a pre-locking process that calls open_and_process_table()
      for each table from the list.
      When a TABLE_LIST for a view is encountered the view is opened and its
      specification is parsed. For any table reference occurred in
      the specification a new TABLE_LIST object is created to be included into
      the list for pre-locking. After all objects in the pre-locking have been
      looked through the tables mentioned in the list are locked. Note that the
      objects referenced CTEs are just skipped here as it is impossible to
      resolve these references without any info on the context where they occur.
      Now the statements from the body of SP are executed one by one that.
      At the very beginning of the execution of a query the tables used in the
      query are opened and open_and_process_table() now is called for each table
      reference mentioned in the list of TABLE_LIST objects associated with the
      query that was built when the query was parsed.
      For each table reference first the reference is checked against CTEs
      definitions in whose scope it occurred. If such definition is found the
      reference is considered resolved and if this is not the first reference
      to the found CTE the the specification of the CTE is re-parsed and the
      result of the parsing is added to the parsing tree of the query as a
      sub-tree. If this sub-tree contains table references to other tables they
      are added to the list of TABLE_LIST objects associated with the query in
      order the referenced tables to be opened. When the procedure that opens
      the tables comes to the TABLE_LIST object created for a non-first
      reference to a CTE it discovers that the referenced table instance is not
      locked and reports an error.
      Thus processing non-first table references to a CTE similar to how
      references to view are processed does not work for queries used in stored
      procedures / functions. And the main problem is that the current
      pre-locking mechanism employed for stored procedures / functions does not
      allow to save the context in which a CTE reference occur. It's not trivial
      to save the info about the context where a CTE reference occurs while the
      resolution of the table reference cannot be done without this context and
      consequentially the specification for the table reference cannot be
      determined.
      
      This patch solves the above problem by moving resolution of all CTE
      references at the parsing stage. More exactly references to CTEs occurred in
      a query are resolved right after parsing of the query has finished. After
      resolution any CTE reference it is marked as a reference to to derived
      table. So it is excluded from the hash table created for pre-locking used
      base tables and view when the first call of a stored procedure / function
      is processed.
      This solution required recursive calls of the parser. The function
      THD::sql_parser() has been added specifically for recursive invocations of
      the parser.
      43c9fcef
    • Marko Mäkelä's avatar
      MDEV-25664 Potential hang in purge for virtual columns · 9739cf18
      Marko Mäkelä authored
      ha_innobase::open(): If the table is only being opened by purge
      for evaluating virtual column values, avoid invoking
      initialize_auto_increment(), because the purge thread may already
      be holding an shared latch on the clustered index root page.
      Shared latches are not recursive. The additional request would lead
      to a hang if another thread has started waiting for an exclusive latch.
      9739cf18
    • Sergei Petrunia's avatar
      MDEV-22462: Item_in_subselect::create_single_in_to_exists_cond(JOIN *, Item... · 2087d47a
      Sergei Petrunia authored
      MDEV-22462: Item_in_subselect::create_single_in_to_exists_cond(JOIN *, Item **, Item **): Assertion `false' failed.
      
      Item_in_subselect::create_single_in_to_exists_cond() should handle the
      case where the subquery is a table-less select but it is not a result
      of a UNION.
      
      (Table-less subqueries like "(SELECT 1)" are "substituted" with their select
      list, but table-less subqueries with WHERE or HAVING clause, like
      "(SELECT 1 WHERE ...)" are not substituted. They are handled with regular
      execution path)
      2087d47a
    • Julius Goryavsky's avatar
      MDEV-25719: stunnel uses "verifyChain" without subject checks · b01a9fd8
      Julius Goryavsky authored
      Another batch of changes that should make the SST process
      more reliable in all scenarios:
      
       1) Added hostname or CN verification when stunnel is used
          with certificate chain verification (verifyChain = yes);
       2) Added check for the absence of the stunnel utility for
          mtr tests;
       3) Deletion of working files before and after SST is done
          more accurately;
       4) rsync on joiner can be run even if the path to its
          configuration file contains spaces;
       5) More accurate directory creation (for data files and
          for logs);
       6) IST with mysqldump no longer turns off statement logging;
       7) Reset password for mysqldump when password is empty but
          username is specified;
       8) More reliable quoting when generating statements in
          wsrep_sst_mysqldump;
       9) Added explicit generation of 2048-bit Diffie-Hellman
          parameters for sockat < 1.7.3, by analogy with xtrabackup;
      10) Compression parameters for qpress are read from all
          suitable server groups in configuration file, as well as
          from the [sst] and [xtrabackup] groups;
      11) Added a test that checks compression using qpress;
      12) Checking for optional utilities is modified to work even
          if they implemented as built-in shell commands (unlikely
          on real systems, but more reliable).
      b01a9fd8
    • Julius Goryavsky's avatar
      MDEV-25719: stunnel uses "verifyChain" without subject checks · 8c8a6ed3
      Julius Goryavsky authored
      Another batch of changes that should make the SST process
      more reliable in all scenarios:
      
       1) Added hostname or CN verification when stunnel is used
          with certificate chain verification (verifyChain = yes);
       2) Added check for the absence of the stunnel utility for
          mtr tests;
       3) Deletion of working files before and after SST is done
          more accurately;
       4) rsync on joiner can be run even if the path to its
          configuration file contains spaces;
       5) More accurate directory creation (for data files and
          for logs);
       6) IST with mysqldump no longer turns off statement logging;
       7) Reset password for mysqldump when password is empty but
          username is specified;
       8) More reliable quoting when generating statements in
          wsrep_sst_mysqldump;
       9) Added explicit generation of 2048-bit Diffie-Hellman
          parameters for sockat < 1.7.3, by analogy with xtrabackup;
      10) Compression parameters for qpress are read from all
          suitable server groups in configuration file, as well as
          from the [sst] and [xtrabackup] groups;
      11) Added a test that checks compression using qpress;
      12) Checking for optional utilities is modified to work even
          if they implemented as built-in shell commands (unlikely
          on real systems, but more reliable).
      8c8a6ed3
  14. 20 May, 2021 1 commit
    • Rucha Deodhar's avatar
      MDEV-25462: Assertion `m_status == DA_ERROR || m_status == DA_OK || · 62944917
      Rucha Deodhar authored
      m_status == DA_OK_BULK' failed in Diagnostics_area::message from
      get_schema_tables_record
      
      Analysis: SET NAMES changes character set for character_set_client,
      character_set_connection, character_set_results to 'filename'. The .frm file of view
      has @xx sequences in the SELECT query, which give parsing error because 'filename'
      character set is not parser friendly. When we get parsing error (ER_PARSE_ERROR), we
      directly return true without setting error status. This is caught later in assertion.
      Fix: Disallow 'filename' character set in SET NAMES because it is not parser
      friendly.
      62944917
  15. 19 May, 2021 3 commits