1. 18 May, 2021 5 commits
    • Marko Mäkelä's avatar
      MDEV-25594: Improve debug checks · 7b51d11c
      Marko Mäkelä authored
      trx_t::will_lock: Changed the type to bool.
      
      trx_t::is_autocommit_non_locking(): Replaces
      trx_is_autocommit_non_locking().
      
      trx_is_ac_nl_ro(): Remove (replaced with equivalent assertion expressions).
      
      assert_trx_nonlocking_or_in_list(): Remove.
      Replaced with at least as strict checks in each place.
      
      check_trx_state(): Moved to a static function; partially replaced with
      individual debug assertions implementing equivalent or stricter checks.
      7b51d11c
    • Marko Mäkelä's avatar
      Merge 10.4 into 10.5 · cc2651b7
      Marko Mäkelä authored
      cc2651b7
    • Marko Mäkelä's avatar
      Merge 10.3 into 10.4 · 4240704a
      Marko Mäkelä authored
      4240704a
    • Marko Mäkelä's avatar
      Merge 10.2 into 10.3, except MDEV-25682 · ca3f4975
      Marko Mäkelä authored
      ca3f4975
    • Marko Mäkelä's avatar
      MDEV-25594: Assertion failure in DeadlockChecker::check_and_resolve() · b9a2e460
      Marko Mäkelä authored
      ha_innobase::index_read(): If an autocommit non-locking transaction was
      already started, refuse to access a SPATIAL INDEX.
      Once a non-locking autocommit transaction has started, it must remain
      in that mode (not acquire any locks).
      
      This should fix one cause of the assertion failure that would occur in
      DeadlockChecker::check_and_resolve() under heavy load, presumably
      due to concurrent execution of trx_commit_in_memory().
      b9a2e460
  2. 17 May, 2021 19 commits
    • Brandon Nesterenko's avatar
      MDEV-25222: mysqlbinlog --base64-output wrong option default drops BINLOG from output · 81402c13
      Brandon Nesterenko authored
      Problem:
      =======
      The ALWAYS option of the mariadb-binlog --base64-output flag
      formats its output incorrectly. This option is deprecated, and
      MySQL 8.0 has removed it entirely.
      
      Solution:
      ========
      Adhere to MySQL and remove this option from MariaDB.
      
      Behavioral Changes:
      ==================
      Use Case: ./mariadb-binlog --base64-output
       Previous Behavior: Sets base64-output mode to always
       New Behavior: Error message indicating incomplete argument
      
      Use Case: ./mariadb-binlog --base64-output=always
       Previous Behavior: Sets base64-output mode to always
       New Behavior: Error message indicating invalid argument value
      
      Reviewed By:
      ==========
      Andrei Elkin: <andrei.elkin@mariadb.com>
      81402c13
    • Julius Goryavsky's avatar
      MDEV-25693: SST failed due to incorrect connection address · 74091762
      Julius Goryavsky authored
      Fixed bugs caused by inaccuracies in automatic merging
      from other branches:
      
      1) Authentication information is not removed from the connection
         address, which causes some tests to fail;
      2) wsrep_debug=on should be replaced with wsrep_debug=1;
      3) Added missing "connection" lines to test result file;
      4) Some tests have been corrected for Galera 4.x (10.4+).
      74091762
    • Julius Goryavsky's avatar
      2947cf64
    • Julius Goryavsky's avatar
      MDEV-25669: SST scripts should check all server groups in config files · 527675d5
      Julius Goryavsky authored
      1) This commit implements reading all sections from configuration
      files while looking for the current value of any server variable,
      which were previously only read from the [mysqld.suffix] group and
      from [mysqld], but not from other groups such as [mariadb.suffix],
      [mariadb] or, for example, [server].
      
      2) This commit also fixes misrecognition of some parameters when
      parsing a command line containing a special marker for the end
      of the list of options ("--") or when short option names (such
      as "-s", "-a" and "-h arg") chained together (like a "-sah arg").
      Such parameters can be passed to the SST script in the list of
      arguments after "--mysqld-args" if the server is started with a
      complex set of options - this was revealed during manual testing
      of changes to read configuration files.
      
      3) The server-side preparation code for the "--mysqld-args"
      option list has also been simplified to make it easier to change
      in the future (if needed), and has been improved to properly
      handle the special backquote ("`") character in the argument
      values.
      527675d5
    • Julius Goryavsky's avatar
    • Julius Goryavsky's avatar
      MDEV-23580: WSREP_SST: [ERROR] rsync daemon port has been taken · d57e60d7
      Julius Goryavsky authored
      This commit contains a large set of further bug fixes and
      improvements to SST scripts for Galera, continuing the work
      that was started in MDEV-24962 to make SST scripts work smoothly
      in different network configurations (especially using ipv6) and
      with different environment settings:
      
       1) The ipv6 addresses were incorrectly handled in the SST script
          for rsync (incorrect address substitution for establishing a
          connection, incorrect address substitution for bind, and so on);
       2) Checking the locality of the ip-address in SST scripts did not
          support ipv6 addresses (such as "[::1]"), which were falsely
          identified as non-local ip, which further did not allow running
          two SSTs on different local addresses on the same machine.
          On the other hand, this bug masked some other errors (related
          to handling ipv6 addresses);
       3) The code for checking the locality of the ip address was different
          in the SST scripts for rsync and for mysqldump, with individual
          flaws. This code is now made common and moved to wsrep_sst_common;
       4) Waiting for the start of the transport channel (socat, nc, rsync,
          stunnel) in the wait_for_listen() and check_pid_and_port() functions
          did not process ipv6 addresses correctly in all cases (not for all
          branches);
       5) Waiting for the start of the transport channel (socat, nc, rsync,
          stunnel) in the wait_for_listen() and check_pid_and_port() functions
          for some code branches could give a false positive result due to
          the textual match of prefixes in the port number and/or PID of
          the process;
       6) Waiting for the start of the transport channel (socat, nc, rsync,
          stunnel) was supported through different utilities in SST scripts
          for mariabackup and for rsync, and with various minor flaws in
          the code. Now the code is still different in these scripts, but
          it supports a common set of utilities (lsof, ss, sockstat) and
          is synchronized across patterns that used to check the output
          of  these utilities;
       7) In SST via mariabackup, the signal about readiness to receive data
          is sometimes sent too early - immediately after listen(), and not
          after accept() (which are called by socat or netcat utility).
       8) Checking availability of the some options of some utilities was
          done using the grep pattern, which easily gives false positives;
       9) Common name (CN) for local addresses, if not explicitly specified,
          is now always replaced to "localhost" to avoid the need to generate
          many separate certificates for local addresses of one machine and
          not to depend on which the local address is currently used in test
          (ipv4 or ipv6, etc.);
      10) In tests galera_sst_mariabackup_encrypt_with_key_server and
          galera_sst_rsync_encrypt_with_key_server the correct certificate
          is selected to avoid commonname (CN) mismatch problems;
      11) Further refactoring to protect against spaces in file names.
      12) Further general refactoring to eliminate bash-specific constructs
          or to improve code readability;
      13) The code for setting options for the nc (netcat) utility was
          different in different scripts for SST - now it is made identical.
      14) Fixed long-time broken encryption via xbcrypt in combination with
          mariabackup and added support for key-based encryption via openssl
          utility, which is now enabled by default for encrypt=1 mode (this
          default mode can be changed using a new configuration file option
          "encypt-format=openssl|xbcrypt", which can be placed in the [mysqld],
          [sst] or in the [xtrabackup] section) - this change will allow us
          to use and to test the encypt=1 encryption without installing
          non-standard third-party utilities.
      d57e60d7
    • Julius Goryavsky's avatar
      MDEV-25693: SST failed due to incorrect connection address · e861e057
      Julius Goryavsky authored
      Fixed bugs caused by inaccuracies in automatic merging
      from other branches:
      
      1) Authentication information is not removed from the connection
         address, which causes some tests to fail;
      2) wsrep_debug=on should be replaced with wsrep_debug=1;
      3) Added missing "connection" lines to test result file;
      4) Some tests have been corrected for Galera 4.x (10.4+).
      e861e057
    • Julius Goryavsky's avatar
      cf4dd3cc
    • Julius Goryavsky's avatar
      9f03a394
    • Julius Goryavsky's avatar
      23cad4d8
    • Julius Goryavsky's avatar
      MDEV-25669: SST scripts should check all server groups in config files · f92cd0c5
      Julius Goryavsky authored
      1) This commit implements reading all sections from configuration
      files while looking for the current value of any server variable,
      which were previously only read from the [mysqld.suffix] group and
      from [mysqld], but not from other groups such as [mariadb.suffix],
      [mariadb] or, for example, [server].
      
      2) This commit also fixes misrecognition of some parameters when
      parsing a command line containing a special marker for the end
      of the list of options ("--") or when short option names (such
      as "-s", "-a" and "-h arg") chained together (like a "-sah arg").
      Such parameters can be passed to the SST script in the list of
      arguments after "--mysqld-args" if the server is started with a
      complex set of options - this was revealed during manual testing
      of changes to read configuration files.
      
      3) The server-side preparation code for the "--mysqld-args"
      option list has also been simplified to make it easier to change
      in the future (if needed), and has been improved to properly
      handle the special backquote ("`") character in the argument
      values.
      f92cd0c5
    • Julius Goryavsky's avatar
      16437e5e
    • Julius Goryavsky's avatar
      MDEV-25669: SST scripts should check all server groups in config files · f9f8e33f
      Julius Goryavsky authored
      1) This commit implements reading all sections from configuration
      files while looking for the current value of any server variable,
      which were previously only read from the [mysqld.suffix] group and
      from [mysqld], but not from other groups such as [mariadb.suffix],
      [mariadb] or, for example, [server].
      
      2) This commit also fixes misrecognition of some parameters when
      parsing a command line containing a special marker for the end
      of the list of options ("--") or when short option names (such
      as "-s", "-a" and "-h arg") chained together (like a "-sah arg").
      Such parameters can be passed to the SST script in the list of
      arguments after "--mysqld-args" if the server is started with a
      complex set of options - this was revealed during manual testing
      of changes to read configuration files.
      
      3) The server-side preparation code for the "--mysqld-args"
      option list has also been simplified to make it easier to change
      in the future (if needed), and has been improved to properly
      handle the special backquote ("`") character in the argument
      values.
      f9f8e33f
    • Julius Goryavsky's avatar
      16898e7f
    • Julius Goryavsky's avatar
    • Julius Goryavsky's avatar
      MDEV-23580: WSREP_SST: [ERROR] rsync daemon port has been taken · 27ae7f2a
      Julius Goryavsky authored
      This commit contains a large set of further bug fixes and
      improvements to SST scripts for Galera, continuing the work
      that was started in MDEV-24962 to make SST scripts work smoothly
      in different network configurations (especially using ipv6) and
      with different environment settings:
      
       1) The ipv6 addresses were incorrectly handled in the SST script
          for rsync (incorrect address substitution for establishing a
          connection, incorrect address substitution for bind, and so on);
       2) Checking the locality of the ip-address in SST scripts did not
          support ipv6 addresses (such as "[::1]"), which were falsely
          identified as non-local ip, which further did not allow running
          two SSTs on different local addresses on the same machine.
          On the other hand, this bug masked some other errors (related
          to handling ipv6 addresses);
       3) The code for checking the locality of the ip address was different
          in the SST scripts for rsync and for mysqldump, with individual
          flaws. This code is now made common and moved to wsrep_sst_common;
       4) Waiting for the start of the transport channel (socat, nc, rsync,
          stunnel) in the wait_for_listen() and check_pid_and_port() functions
          did not process ipv6 addresses correctly in all cases (not for all
          branches);
       5) Waiting for the start of the transport channel (socat, nc, rsync,
          stunnel) in the wait_for_listen() and check_pid_and_port() functions
          for some code branches could give a false positive result due to
          the textual match of prefixes in the port number and/or PID of
          the process;
       6) Waiting for the start of the transport channel (socat, nc, rsync,
          stunnel) was supported through different utilities in SST scripts
          for mariabackup and for rsync, and with various minor flaws in
          the code. Now the code is still different in these scripts, but
          it supports a common set of utilities (lsof, ss, sockstat) and
          is synchronized across patterns that used to check the output
          of  these utilities;
       7) In SST via mariabackup, the signal about readiness to receive data
          is sometimes sent too early - immediately after listen(), and not
          after accept() (which are called by socat or netcat utility).
       8) Checking availability of the some options of some utilities was
          done using the grep pattern, which easily gives false positives;
       9) Common name (CN) for local addresses, if not explicitly specified,
          is now always replaced to "localhost" to avoid the need to generate
          many separate certificates for local addresses of one machine and
          not to depend on which the local address is currently used in test
          (ipv4 or ipv6, etc.);
      10) In tests galera_sst_mariabackup_encrypt_with_key_server and
          galera_sst_rsync_encrypt_with_key_server the correct certificate
          is selected to avoid commonname (CN) mismatch problems;
      11) Further refactoring to protect against spaces in file names.
      12) Further general refactoring to eliminate bash-specific constructs
          or to improve code readability;
      13) The code for setting options for the nc (netcat) utility was
          different in different scripts for SST - now it is made identical.
      14) Fixed long-time broken encryption via xbcrypt in combination with
          mariabackup and added support for key-based encryption via openssl
          utility, which is now enabled by default for encrypt=1 mode (this
          default mode can be changed using a new configuration file option
          "encypt-format=openssl|xbcrypt", which can be placed in the [mysqld],
          [sst] or in the [xtrabackup] section) - this change will allow us
          to use and to test the encypt=1 encryption without installing
          non-standard third-party utilities.
      27ae7f2a
    • Sujatha's avatar
      MDEV-22530: Aborting OPTIMIZE TABLE still logs in binary log and replicates to the Slave server. · 88c7a58e
      Sujatha authored
      Problem:
      ========
      Aborting OPTIMIZE TABLE still logs in binary logs and replicates to the
      Slave server. "Optimize table" command under execution, is killed by using
      "Ctrl-C" as shown below.
      
      MariaDB [test]> optimize table t2;
      ^CCtrl-C -- query killed. Continuing normally.
      
      In spite of query execution being interrupted the query gets written to
      binary log.
      
      Analysis:
      ========
      Admin command execution logic is not handling KILL command, hence it
      ignores the KILL command and completes its execution.
      
      Fix:
      ===
      Check for thread killed notification, during admin command execution and
      handle it. If thread kill occurs prior to any table modification the query
      will not be written to binary log. If kill happens after at least one table
      is modified then the query will be written to binary log. Ex: command in
      execution is 'OPTIMIZE TABLE t1,t2' and the thread kill happens after t1
      table is modified then 'OPTIMIZE TABLE t1,t2' will be written to binary log
      as admin commands will not make the slave to diverge from master.
      88c7a58e
    • Sujatha's avatar
      MDEV-17515: GTID Replication in optimistic mode deadlock · 410e3c1a
      Sujatha authored
      Problem:
      =======
      In slave_parallel_mode=optimistic configuration, when admin commands and
      DML operation on the same table are scheduled simultaneously for execution,
      it results in lock conflict and slave server either hangs due to
      deadlock or goes down with an assert.
      
      Analysis:
      ========
      Admin commands OPTIMIZE, REPAIR and ANALYZE are written to binary log as
      ordinary transactions. When 'slave_parallel_mode' is 'optimistic' DMLs are
      allowed to run in parallel. But these locks are not detected by parallel
      replication deadlock detection-and-handling mechanism. At times they result
      in deadlock or assertion.
      
      Fix:
      ===
      Flag admin commands as DDL in Gtid_log_event at the time of writing to
      binary log. Add a new bit EXECUTED_TABLE_ADMIN_CMD to
      'm_unsafe_rollback_flags'. During 'mysql_admin_table' command execution it
      accepts a list of tables to be processed and executes them in a loop. Upon
      successful execution enable 'EXECUTED_TABLE_ADMIN_CMD' bit in
      thd->transaction.stmt_unsafe_rollback_flags. Gtid_log_event constructor
      will notice this flag and mark the current transaction with 'FL_DDL' flag.
      Gtid_log_events marked as FL_DDL will not be scheduled parallel execution,
      on the slave. They will execute in isolation to prevent deadlocks.
      
      Note: Removed the call to 'trans_commit_implicit' from 'mysql_admin_table'
      function as 'mysql_execute_command' will take care of invoking
      'trans_commit_implicit'.
      410e3c1a
    • Alexey Botchkov's avatar
      MDEV-25690 Plugins can't execute sql statements with the Galera enabled. · c22d567e
      Alexey Botchkov authored
      For the plugin thread during the server start the wsrep_on should be 0.
      c22d567e
  3. 16 May, 2021 1 commit
  4. 15 May, 2021 2 commits
    • Julius Goryavsky's avatar
      MDEV-25669: SST scripts should check all server groups in config files · 6811ed3e
      Julius Goryavsky authored
      1) This commit implements reading all sections from configuration
      files while looking for the current value of any server variable,
      which were previously only read from the [mysqld.suffix] group and
      from [mysqld], but not from other groups such as [mariadb.suffix],
      [mariadb] or, for example, [server].
      
      2) This commit also fixes misrecognition of some parameters when
      parsing a command line containing a special marker for the end
      of the list of options ("--") or when short option names (such
      as "-s", "-a" and "-h arg") chained together (like a "-sah arg").
      Such parameters can be passed to the SST script in the list of
      arguments after "--mysqld-args" if the server is started with a
      complex set of options - this was revealed during manual testing
      of changes to read configuration files.
      
      3) The server-side preparation code for the "--mysqld-args"
      option list has also been simplified to make it easier to change
      in the future (if needed), and has been improved to properly
      handle the special backquote ("`") character in the argument
      values.
      6811ed3e
    • Julius Goryavsky's avatar
      4675febb
  5. 14 May, 2021 4 commits
    • Igor Babaev's avatar
      MDEV-25682 Explain shows an execution plan different from actually executed · 677f1ef6
      Igor Babaev authored
      If a select query contained an ORDER BY clause that followed a LIMIT clause
      or an ORDER BY clause or ORDER BY with LIMIT the EXPLAIN output for the
      query showed an execution plan different from that was actually executed.
      
      Approved by Roman Nozdrin <roman.nozdrin@mariadb.com>
      677f1ef6
    • Sachin Kumar's avatar
      MDEV-25336 Parallel replication causes failed assert while restarting · e607f339
      Sachin Kumar authored
      Problem:- When slave is shutdown, we will get this assertion failure
      sql/sql_list.h:642: void ilink::assert_linked(): Assertion `prev != 0
      && next != 0' failed.
      
      Solution:- In close_connections when we call threads.get() it resets to
      prev and next to NULL. And in parallel worker thread(handle_rpl_parallel_thread)
      calls unlink_not_visible_thd() which assert on prev and next being not NULL.
      .unlink_not_visible_thd() should be always called first before threads.get()
      is called. To make sure worker calls unlink_not_visible_thd() in
      slave_prepare_for_shutdown() we are deactivating the  worker thread pool
      which in turn will close all worker threads. Since this is already done in 10.4
      and 10.5 I am backPorting MDEV-20821 and MDEV-22370 to 10.2. Mdev-22370
      is improving the MDEV-20821 patch.
      e607f339
    • Sachin Kumar's avatar
      MDEV-22370 safe_mutex: Trying to lock uninitialized mutex at... · 355dc74b
      Sachin Kumar authored
      MDEV-22370 safe_mutex: Trying to lock uninitialized mutex at /data/src/10.4-bug/sql/rpl_parallel.cc, line 470 upon shutdown during FTWRL
      
      Problem:- When we issue FTWRL with shutdown in parallel, there is race between
      FTWRL and shutdown. Shutdown might destroy the mutex (pool->LOCK_rpl_thread_pool)
      before FTWRL can lock it. So we can get crash on FTWRL thread
      
      Solution:- mysql_mutex_destroy(pool->LOCK_rpl_thread_pool) should wait for
      FTWRL thread to complete its work , and then destroy.
      So slave_prepare_for_shutdown will just deactivate the pool, and mutex is destroyed
      later in end_slave()
      355dc74b
    • Andrei Elkin's avatar
      MDEV-20821 parallel slave server shutdown hang · 3616640a
      Andrei Elkin authored
      Parallel slave server shutdown found to be hanging in
      close_connections() triggered by shutdown due to a slave worker thread
      would not be notified to exit in case the worker was sitting idle.
      
      Fixed with destroying the worker pool earlier that is in
      slave_prepare_for_shutdown() when all their driver threads have already left.
      A test file is added to simulate the bug condition as well as check
      multi-sourced and not-idle worker cases.
      3616640a
  6. 11 May, 2021 7 commits
    • Monty's avatar
      4d53a758
    • Sergei Golubchik's avatar
      ec348f55
    • Monty's avatar
      MDEV-25651 Server crash or assertion failure in THD::update_stats upon concurrent DROP TRIGGER · 0df51e61
      Monty authored
      The problem was that in a timeout event,
      thd->lex->restore_backup_query_tables_list() was called when it should
      not have been.
      
      Patch tested with the script in MDEV-25651 (not suitable for mtr)
      0df51e61
    • Ramesh Sivaraman's avatar
      Updated galera_3nodes disabled.def file · 1447b394
      Ramesh Sivaraman authored
      1447b394
    • Julius Goryavsky's avatar
    • Julius Goryavsky's avatar
    • Julius Goryavsky's avatar
      MDEV-23580: WSREP_SST: [ERROR] rsync daemon port has been taken · 7e8a8938
      Julius Goryavsky authored
      This commit contains a large set of further bug fixes and
      improvements to SST scripts for Galera, continuing the work
      that was started in MDEV-24962 to make SST scripts work smoothly
      in different network configurations (especially using ipv6) and
      with different environment settings:
      
       1) The ipv6 addresses were incorrectly handled in the SST script
          for rsync (incorrect address substitution for establishing a
          connection, incorrect address substitution for bind, and so on);
       2) Checking the locality of the ip-address in SST scripts did not
          support ipv6 addresses (such as "[::1]"), which were falsely
          identified as non-local ip, which further did not allow running
          two SSTs on different local addresses on the same machine.
          On the other hand, this bug masked some other errors (related
          to handling ipv6 addresses);
       3) The code for checking the locality of the ip address was different
          in the SST scripts for rsync and for mysqldump, with individual
          flaws. This code is now made common and moved to wsrep_sst_common;
       4) Waiting for the start of the transport channel (socat, nc, rsync,
          stunnel) in the wait_for_listen() and check_pid_and_port() functions
          did not process ipv6 addresses correctly in all cases (not for all
          branches);
       5) Waiting for the start of the transport channel (socat, nc, rsync,
          stunnel) in the wait_for_listen() and check_pid_and_port() functions
          for some code branches could give a false positive result due to
          the textual match of prefixes in the port number and/or PID of
          the process;
       6) Waiting for the start of the transport channel (socat, nc, rsync,
          stunnel) was supported through different utilities in SST scripts
          for mariabackup and for rsync, and with various minor flaws in
          the code. Now the code is still different in these scripts, but
          it supports a common set of utilities (lsof, ss, sockstat) and
          is synchronized across patterns that used to check the output
          of  these utilities;
       7) In SST via mariabackup, the signal about readiness to receive data
          is sometimes sent too early - immediately after listen(), and not
          after accept() (which are called by socat or netcat utility).
       8) Checking availability of the some options of some utilities was
          done using the grep pattern, which easily gives false positives;
       9) Common name (CN) for local addresses, if not explicitly specified,
          is now always replaced to "localhost" to avoid the need to generate
          many separate certificates for local addresses of one machine and
          not to depend on which the local address is currently used in test
          (ipv4 or ipv6, etc.);
      10) In tests galera_sst_mariabackup_encrypt_with_key_server and
          galera_sst_rsync_encrypt_with_key_server the correct certificate
          is selected to avoid commonname (CN) mismatch problems;
      11) Further refactoring to protect against spaces in file names.
      12) Further general refactoring to eliminate bash-specific constructs
          or to improve code readability;
      13) The code for setting options for the nc (netcat) utility was
          different in different scripts for SST - now it is made identical.
      14) Fixed long-time broken encryption via xbcrypt in combination with
          mariabackup and added support for key-based encryption via openssl
          utility, which is now enabled by default for encrypt=1 mode (this
          default mode can be changed using a new configuration file option
          "encypt-format=openssl|xbcrypt", which can be placed in the [mysqld],
          [sst] or in the [xtrabackup] section) - this change will allow us
          to use and to test the encypt=1 encryption without installing
          non-standard third-party utilities.
      7e8a8938
  7. 10 May, 2021 2 commits