1. 19 May, 2021 20 commits
    • Monty's avatar
      Renamed Type_all_attributes::set_maybe_null() to set_type_maybe_null() · cd1782d2
      Monty authored
      This was done to have Item::set_maybe_null() as an inline function
      cd1782d2
    • Monty's avatar
      c6bf04ab
    • Monty's avatar
      Removed not used Item_type_holder() function · 61822a9b
      Monty authored
      61822a9b
    • Monty's avatar
      Enable BUILD scripts to work with clang · 60335ba1
      Monty authored
      60335ba1
    • Monty's avatar
    • Monty's avatar
      Fixes that enables my_new.cc (new wrapper using my_malloc) · e42130e9
      Monty authored
      This is not enabled by default, as there are leaks in the
      server that needs to be fixed first. One can compile
      with -DUSE_MYSYS_NEW to find the memory leaks from
      'new'. More comments can be found in mysys/my_new.cc
      e42130e9
    • Monty's avatar
      Fixed my_addr_resolve · a93c5145
      Monty authored
      When a server is compiled with -fPIE, my_addr_resolve needs to subtract the info.dli_fbase from symbol addresses in memory for addr2line to recognize them.
      When a server is compiled without -fPIE, my_addr_resolve should not do it.
      Unfortunately not all compilers define __PIE__ when -fPIE was used
      (e.g. older gcc doesn't), so we have to resort to run-time detection.
      a93c5145
    • Monty's avatar
      Report memory leaks from mariadbd if -T or --debug is used · 942a5a89
      Monty authored
      Before memory leaks was only reported if server stopped normally.
      This made it harder to find out where the leaks happened when
      debugging test cases.
      942a5a89
    • Monty's avatar
      Optimize usage of c_ptr(), c_ptr_quick() and String::alloc() · 36cdd5c3
      Monty authored
      The problem was that when one used String::alloc() to allocate a string,
      the String ensures that there is space for an extra NULL byte in the
      buffer and if not, reallocates the string. This is a problem with the
      String::set_int() that calls alloc(21), which forces extra
      malloc/free calls to happen.
      
      - We do not anymore re-allocate String if alloc() is called with the
        Allocated_length. This reduces number of malloc() allocations,
        especially one big re-allocation in Protocol::send_result_Set_metadata()
        for almost every query that produced a result to the connnected client.
      - Avoid extra mallocs when using LONGLONG_BUFFER_SIZE
        This can now be done as alloc() doesn't increase buffers if new length is
        not bigger than old one.
      - c_ptr() is redesigned to be safer (but a bit longer) than before.
      - Remove wrong usage of c_ptr_quick()
        c_ptr_quick() was used in many cases to get the pointer to the used
        buffer, even when it didn't need to be \0 terminated. In this case
        ptr() is a better substitute.
        Another problem with c_ptr_quick() is that it did not guarantee that
        the string would be \0 terminated.
      - item_val_str(), an API function not used currently by the server,
        now always returns a null terminated string (before it didn't always
        do that).
      - Ensure that all String allocations uses STRING_PSI_MEMORY_KEY. The old
        mixed usage of performance keys caused assert's when String buffers
        where shrunk.
      - Binary_string::shrink() is simplifed
      - Fixed bug in String(const char *str, size_t len, CHARSET_INFO *cs) that
        used Binary_string((char *) str, len) instead of Binary_string(str,len).
      - Changed argument to String() creations and String.set() functions to use
        'const char*' instead of 'char*'. This ensures that Alloced_length is
        not set, which gives safety against someone trying to change the
        original string. This also would allow us to use !Alloced_length in
        c_ptr() if needed.
      - Changed string_ptr_cmp() to use memcmp() instead of c_ptr() to avoid
        a possible malloc during string comparision.
      36cdd5c3
    • Monty's avatar
      Optimize Sql_alloc · da85ad79
      Monty authored
      - Remove 'dummy_for_valgrind' overrun marker as this doesn't help much.
        The element also distorts the sizes of objects a bit, which makes it
        harder to calculate gain in object sizes when doing size optimizations.
      - Replace usage of thd_get_current_thd() with _current_thd()
      - Avoid one extra call indirection when using thd_get_current_thd(), which
        is used by Sql_alloc, by replacing it with _current_thd()
      da85ad79
    • Monty's avatar
      Improved storage size for Item, Field and some other classes · c76eabfb
      Monty authored
      - Changed order of class fields to remove dead alignment space.
      - Changed bool fields in Item to bit fields.
      - Used packed enum's for some fields in common classes
      - Removed not used Item::rsize.
      - Changed some class variables from uint/int to smaller type int's.
      - Ensured that field_index is uint16 in all classes and functions. Fixed
        also that we proparly compare with NO_CACHED_FIELD_INDEX when checking
        if variable is not set.
      - Removed checking of highest bit of unireg_check (has not been used in
        a long time)
      - Fixed wrong arguments to make_cond_for_table() for join_tab_idx_arg
        from false to 0.
      
      One of the result was reducing the size if class Item with ~24 bytes
      c76eabfb
    • Monty's avatar
      Optimize size of lex structures · 8e8bda7f
      Monty authored
      LEX, st_select_lex, st_select_unit optimized for space:
      - Use bit fields for bool variables
      - Ensure that all bit fields are initialized (improves
        performance for init functions as all bit fields can be
        initalized with one memory access)
      - Move members around in above structures to remove alignment
        gaps
      
      Some savings:
      LEX: 7032 -> 6880
      THD: 25608 -> 25456
      st_select_lex_unit: 2048 -> 2008
      
      LEX::start():                    1321 -> 1245 instructions
      st_select_lex_unit::init_query()  284 ->  214 instructions
      st_select_lex::init_query():      766 ->  692 instructions
      st_select_lex::init_select():     563 ->  540 instructions
      
      Other things:
      - Removed not used LEX::select_allow_into
      - Fixed MDEV-25510 Assertion `sel->select_lock ==
         st_select_lex::select_lock_type::NONE' which was caused by this commit.
      8e8bda7f
    • Monty's avatar
      Added typedef decimal_digits_t (uint16) for number of digits in most · fa7d4abf
      Monty authored
      aspects of decimals and integers
      
      For fields and Item's uint8 should be good enough. After
      discussions with Alexander Barkov we choose uint16 (for now)
      as some format functions may accept +256 digits.
      
      The reason for this patch was to make the usage and storage of decimal
      digits simlar. Before this patch decimals was stored/used as uint8,
      int and uint.  The lengths for numbers where also using a lot of
      different types.
      
      Changed most decimal variables and functions to use the new typedef.
      
      squash! af7f09106b6c1dc20ae8c480bff6fd22d266b184
      
      Use decimal_digits_t for all aspects of digits (total, precision
      and scale), both for decimals and integers.
      fa7d4abf
    • Monty's avatar
      MDEV-23001 Precreate static Item_bool() to simplify code · aee84453
      Monty authored
      The following changes where done:
      - Create global Item: Item_false and Item_true
      - Replace all creation if 'FALSE' and 'TRUE' top level items used for
        WHERE/HAVING/ON clauses to use Item_false and Item_true.
      
      The benefit are:
      - Less and faster code
      - No test needed if we where able to create the new item.
      - Fixed possible errors if 'new' would have failed for the Item_bool's
      
      fixup! 470277728d2e27fe057cf33a437a9e40e1a04b61
      aee84453
    • Brandon Nesterenko's avatar
      MDEV-25222: mysqlbinlog --base64-output wrong option default drops BINLOG from output · 78a0fe79
      Brandon Nesterenko authored
      The --help comment for the --base64-output option in
      mysqlbinlog was hard to decipher. This quick patch aims
      to refine it.
      
      Reviewed By:
      ==========
      Andrei Elkin: <andrei.elkin@mariadb.com>
      78a0fe79
    • Marko Mäkelä's avatar
      MDEV-25691 fixup: Correctly drop orphan foreign keys · 27141581
      Marko Mäkelä authored
      innodb_drop_database_fk(): Use the correct length when comparing.
      Fix a debug assertion in previously unreachable code.
      This error was caught by MSAN.
      
      innodb_drop_database(): Correct the SQL for traversing SYS_FOREIGN.
      The incorrect code would cause orphan FOREIGN KEY entries to be
      left behind in the test innodb.alter_foreign_crash.
      27141581
    • Marko Mäkelä's avatar
      Merge 10.5 into 10.6 · 5d495fc4
      Marko Mäkelä authored
      5d495fc4
    • Marko Mäkelä's avatar
      MDEV-8334: Adjust test results · e3adb434
      Marko Mäkelä authored
      e3adb434
    • Marko Mäkelä's avatar
      Merge 10.4 into 10.5 · db8fb408
      Marko Mäkelä authored
      db8fb408
    • Rucha Deodhar's avatar
      MDEV-8334: Rename utf8 to utf8mb3 · 2fdb556e
      Rucha Deodhar authored
      This patch changes the main name of 3 byte character set from utf8 to
      utf8mb3. New old_mode UTF8_IS_UTF8MB3 is added and set TRUE by default,
      so that utf8 would mean utf8mb3. If not set, utf8 would mean utf8mb4.
      2fdb556e
  2. 18 May, 2021 9 commits
    • Marko Mäkelä's avatar
    • Marko Mäkelä's avatar
      MDEV-25691: Simplify handlerton::drop_database for InnoDB · c366845a
      Marko Mäkelä authored
      The implementation of handlerton::drop_database in InnoDB is
      unnecessarily complex. The minimal implementation should check
      that no conflicting locks or references exist on the tables,
      delete all table metadata in a single transaction, and finally
      delete the tablespaces.
      
      Note: DROP DATABASE will delete each individual table that the
      SQL layer knows about, one table per transaction.
      The handlerton::drop_database is basically a final cleanup step
      for removing any garbage that could have been left behind
      in InnoDB due to some bug, or not having atomic DDL in the past.
      
      hash_node_t: Remove. Use the proper data type name in pointers.
      
      dict_drop_index_tree(): Do not take the table as a parameter.
      Instead, return the tablespace ID if the tablespace should be dropped
      (we are dropping a clustered index tree).
      
      fil_delete_tablespace(), fil_system_t::detach(): Return a single
      detached file handle. Multi-file tablespaces cannot be deleted
      via this interface.
      
      ha_innobase::delete_table(): Remove a work-around for non-atomic DDL
      and do not try to drop tables with similar-looking name.
      
      innodb_drop_database(): Complete rewrite.
      
      innobase_drop_database(), dict_get_first_table_name_in_db(),
      row_drop_database_for_mysql(), drop_all_foreign_keys_in_db(): Remove.
      
      row_purge_remove_clust_if_poss_low(), row_undo_ins_remove_clust_rec():
      If the tablespace is to be deleted, try to evict the table definition
      from the cache. Failing that, set dict_table_t::space to nullptr.
      
      lock_release_on_rollback(): On the rollback of CREATE TABLE, release all
      locks that the transaction had on the table, to avoid heap-use-after-free.
      c366845a
    • Marko Mäkelä's avatar
      MDEV-25710: Dead code os_file_opendir() in the server · 08b6fd93
      Marko Mäkelä authored
      The functions fil_file_readdir_next_file(), os_file_opendir(),
      os_file_closedir() became dead code in the server in MariaDB 10.4.0
      with commit 09af00cb (the removal of
      the crash recovery logic for the TRUNCATE TABLE implementation that
      was replaced in MDEV-13564).
      
      os_file_opendir(), os_file_closedir(): Define as macros.
      08b6fd93
    • Marko Mäkelä's avatar
      Merge 10.5 into 10.6 · f09d33f5
      Marko Mäkelä authored
      f09d33f5
    • Marko Mäkelä's avatar
      MDEV-25594: Improve debug checks · 7b51d11c
      Marko Mäkelä authored
      trx_t::will_lock: Changed the type to bool.
      
      trx_t::is_autocommit_non_locking(): Replaces
      trx_is_autocommit_non_locking().
      
      trx_is_ac_nl_ro(): Remove (replaced with equivalent assertion expressions).
      
      assert_trx_nonlocking_or_in_list(): Remove.
      Replaced with at least as strict checks in each place.
      
      check_trx_state(): Moved to a static function; partially replaced with
      individual debug assertions implementing equivalent or stricter checks.
      7b51d11c
    • Marko Mäkelä's avatar
      Merge 10.4 into 10.5 · cc2651b7
      Marko Mäkelä authored
      cc2651b7
    • Marko Mäkelä's avatar
      Merge 10.3 into 10.4 · 4240704a
      Marko Mäkelä authored
      4240704a
    • Marko Mäkelä's avatar
      Merge 10.2 into 10.3, except MDEV-25682 · ca3f4975
      Marko Mäkelä authored
      ca3f4975
    • Marko Mäkelä's avatar
      MDEV-25594: Assertion failure in DeadlockChecker::check_and_resolve() · b9a2e460
      Marko Mäkelä authored
      ha_innobase::index_read(): If an autocommit non-locking transaction was
      already started, refuse to access a SPATIAL INDEX.
      Once a non-locking autocommit transaction has started, it must remain
      in that mode (not acquire any locks).
      
      This should fix one cause of the assertion failure that would occur in
      DeadlockChecker::check_and_resolve() under heavy load, presumably
      due to concurrent execution of trx_commit_in_memory().
      b9a2e460
  3. 17 May, 2021 11 commits
    • Brandon Nesterenko's avatar
      MDEV-25222: mysqlbinlog --base64-output wrong option default drops BINLOG from output · 81402c13
      Brandon Nesterenko authored
      Problem:
      =======
      The ALWAYS option of the mariadb-binlog --base64-output flag
      formats its output incorrectly. This option is deprecated, and
      MySQL 8.0 has removed it entirely.
      
      Solution:
      ========
      Adhere to MySQL and remove this option from MariaDB.
      
      Behavioral Changes:
      ==================
      Use Case: ./mariadb-binlog --base64-output
       Previous Behavior: Sets base64-output mode to always
       New Behavior: Error message indicating incomplete argument
      
      Use Case: ./mariadb-binlog --base64-output=always
       Previous Behavior: Sets base64-output mode to always
       New Behavior: Error message indicating invalid argument value
      
      Reviewed By:
      ==========
      Andrei Elkin: <andrei.elkin@mariadb.com>
      81402c13
    • Julius Goryavsky's avatar
      MDEV-25693: SST failed due to incorrect connection address · 34340fb5
      Julius Goryavsky authored
      Fixed bugs caused by inaccuracies in automatic merging
      from other branches:
      
      1) Authentication information is not removed from the connection
         address, which causes some tests to fail;
      2) wsrep_debug=on should be replaced with wsrep_debug=1;
      3) Added missing "connection" lines to test result file;
      4) Some tests have been corrected for Galera 4.x (10.4+).
      34340fb5
    • Julius Goryavsky's avatar
      d45f61c3
    • Julius Goryavsky's avatar
      MDEV-25669: SST scripts should check all server groups in config files · 9e8e82cc
      Julius Goryavsky authored
      1) This commit implements reading all sections from configuration
      files while looking for the current value of any server variable,
      which were previously only read from the [mysqld.suffix] group and
      from [mysqld], but not from other groups such as [mariadb.suffix],
      [mariadb] or, for example, [server].
      
      2) This commit also fixes misrecognition of some parameters when
      parsing a command line containing a special marker for the end
      of the list of options ("--") or when short option names (such
      as "-s", "-a" and "-h arg") chained together (like a "-sah arg").
      Such parameters can be passed to the SST script in the list of
      arguments after "--mysqld-args" if the server is started with a
      complex set of options - this was revealed during manual testing
      of changes to read configuration files.
      
      3) The server-side preparation code for the "--mysqld-args"
      option list has also been simplified to make it easier to change
      in the future (if needed), and has been improved to properly
      handle the special backquote ("`") character in the argument
      values.
      9e8e82cc
    • Julius Goryavsky's avatar
    • Julius Goryavsky's avatar
      MDEV-23580: WSREP_SST: [ERROR] rsync daemon port has been taken · bcd6af93
      Julius Goryavsky authored
      This commit contains a large set of further bug fixes and
      improvements to SST scripts for Galera, continuing the work
      that was started in MDEV-24962 to make SST scripts work smoothly
      in different network configurations (especially using ipv6) and
      with different environment settings:
      
       1) The ipv6 addresses were incorrectly handled in the SST script
          for rsync (incorrect address substitution for establishing a
          connection, incorrect address substitution for bind, and so on);
       2) Checking the locality of the ip-address in SST scripts did not
          support ipv6 addresses (such as "[::1]"), which were falsely
          identified as non-local ip, which further did not allow running
          two SSTs on different local addresses on the same machine.
          On the other hand, this bug masked some other errors (related
          to handling ipv6 addresses);
       3) The code for checking the locality of the ip address was different
          in the SST scripts for rsync and for mysqldump, with individual
          flaws. This code is now made common and moved to wsrep_sst_common;
       4) Waiting for the start of the transport channel (socat, nc, rsync,
          stunnel) in the wait_for_listen() and check_pid_and_port() functions
          did not process ipv6 addresses correctly in all cases (not for all
          branches);
       5) Waiting for the start of the transport channel (socat, nc, rsync,
          stunnel) in the wait_for_listen() and check_pid_and_port() functions
          for some code branches could give a false positive result due to
          the textual match of prefixes in the port number and/or PID of
          the process;
       6) Waiting for the start of the transport channel (socat, nc, rsync,
          stunnel) was supported through different utilities in SST scripts
          for mariabackup and for rsync, and with various minor flaws in
          the code. Now the code is still different in these scripts, but
          it supports a common set of utilities (lsof, ss, sockstat) and
          is synchronized across patterns that used to check the output
          of  these utilities;
       7) In SST via mariabackup, the signal about readiness to receive data
          is sometimes sent too early - immediately after listen(), and not
          after accept() (which are called by socat or netcat utility).
       8) Checking availability of the some options of some utilities was
          done using the grep pattern, which easily gives false positives;
       9) Common name (CN) for local addresses, if not explicitly specified,
          is now always replaced to "localhost" to avoid the need to generate
          many separate certificates for local addresses of one machine and
          not to depend on which the local address is currently used in test
          (ipv4 or ipv6, etc.);
      10) In tests galera_sst_mariabackup_encrypt_with_key_server and
          galera_sst_rsync_encrypt_with_key_server the correct certificate
          is selected to avoid commonname (CN) mismatch problems;
      11) Further refactoring to protect against spaces in file names.
      12) Further general refactoring to eliminate bash-specific constructs
          or to improve code readability;
      13) The code for setting options for the nc (netcat) utility was
          different in different scripts for SST - now it is made identical.
      14) Fixed long-time broken encryption via xbcrypt in combination with
          mariabackup and added support for key-based encryption via openssl
          utility, which is now enabled by default for encrypt=1 mode (this
          default mode can be changed using a new configuration file option
          "encypt-format=openssl|xbcrypt", which can be placed in the [mysqld],
          [sst] or in the [xtrabackup] section) - this change will allow us
          to use and to test the encypt=1 encryption without installing
          non-standard third-party utilities.
      bcd6af93
    • Julius Goryavsky's avatar
      MDEV-25693: SST failed due to incorrect connection address · 74091762
      Julius Goryavsky authored
      Fixed bugs caused by inaccuracies in automatic merging
      from other branches:
      
      1) Authentication information is not removed from the connection
         address, which causes some tests to fail;
      2) wsrep_debug=on should be replaced with wsrep_debug=1;
      3) Added missing "connection" lines to test result file;
      4) Some tests have been corrected for Galera 4.x (10.4+).
      74091762
    • Julius Goryavsky's avatar
      2947cf64
    • Julius Goryavsky's avatar
      MDEV-25669: SST scripts should check all server groups in config files · 527675d5
      Julius Goryavsky authored
      1) This commit implements reading all sections from configuration
      files while looking for the current value of any server variable,
      which were previously only read from the [mysqld.suffix] group and
      from [mysqld], but not from other groups such as [mariadb.suffix],
      [mariadb] or, for example, [server].
      
      2) This commit also fixes misrecognition of some parameters when
      parsing a command line containing a special marker for the end
      of the list of options ("--") or when short option names (such
      as "-s", "-a" and "-h arg") chained together (like a "-sah arg").
      Such parameters can be passed to the SST script in the list of
      arguments after "--mysqld-args" if the server is started with a
      complex set of options - this was revealed during manual testing
      of changes to read configuration files.
      
      3) The server-side preparation code for the "--mysqld-args"
      option list has also been simplified to make it easier to change
      in the future (if needed), and has been improved to properly
      handle the special backquote ("`") character in the argument
      values.
      527675d5
    • Julius Goryavsky's avatar
    • Julius Goryavsky's avatar
      MDEV-23580: WSREP_SST: [ERROR] rsync daemon port has been taken · d57e60d7
      Julius Goryavsky authored
      This commit contains a large set of further bug fixes and
      improvements to SST scripts for Galera, continuing the work
      that was started in MDEV-24962 to make SST scripts work smoothly
      in different network configurations (especially using ipv6) and
      with different environment settings:
      
       1) The ipv6 addresses were incorrectly handled in the SST script
          for rsync (incorrect address substitution for establishing a
          connection, incorrect address substitution for bind, and so on);
       2) Checking the locality of the ip-address in SST scripts did not
          support ipv6 addresses (such as "[::1]"), which were falsely
          identified as non-local ip, which further did not allow running
          two SSTs on different local addresses on the same machine.
          On the other hand, this bug masked some other errors (related
          to handling ipv6 addresses);
       3) The code for checking the locality of the ip address was different
          in the SST scripts for rsync and for mysqldump, with individual
          flaws. This code is now made common and moved to wsrep_sst_common;
       4) Waiting for the start of the transport channel (socat, nc, rsync,
          stunnel) in the wait_for_listen() and check_pid_and_port() functions
          did not process ipv6 addresses correctly in all cases (not for all
          branches);
       5) Waiting for the start of the transport channel (socat, nc, rsync,
          stunnel) in the wait_for_listen() and check_pid_and_port() functions
          for some code branches could give a false positive result due to
          the textual match of prefixes in the port number and/or PID of
          the process;
       6) Waiting for the start of the transport channel (socat, nc, rsync,
          stunnel) was supported through different utilities in SST scripts
          for mariabackup and for rsync, and with various minor flaws in
          the code. Now the code is still different in these scripts, but
          it supports a common set of utilities (lsof, ss, sockstat) and
          is synchronized across patterns that used to check the output
          of  these utilities;
       7) In SST via mariabackup, the signal about readiness to receive data
          is sometimes sent too early - immediately after listen(), and not
          after accept() (which are called by socat or netcat utility).
       8) Checking availability of the some options of some utilities was
          done using the grep pattern, which easily gives false positives;
       9) Common name (CN) for local addresses, if not explicitly specified,
          is now always replaced to "localhost" to avoid the need to generate
          many separate certificates for local addresses of one machine and
          not to depend on which the local address is currently used in test
          (ipv4 or ipv6, etc.);
      10) In tests galera_sst_mariabackup_encrypt_with_key_server and
          galera_sst_rsync_encrypt_with_key_server the correct certificate
          is selected to avoid commonname (CN) mismatch problems;
      11) Further refactoring to protect against spaces in file names.
      12) Further general refactoring to eliminate bash-specific constructs
          or to improve code readability;
      13) The code for setting options for the nc (netcat) utility was
          different in different scripts for SST - now it is made identical.
      14) Fixed long-time broken encryption via xbcrypt in combination with
          mariabackup and added support for key-based encryption via openssl
          utility, which is now enabled by default for encrypt=1 mode (this
          default mode can be changed using a new configuration file option
          "encypt-format=openssl|xbcrypt", which can be placed in the [mysqld],
          [sst] or in the [xtrabackup] section) - this change will allow us
          to use and to test the encypt=1 encryption without installing
          non-standard third-party utilities.
      d57e60d7