Some users and some scripts (for example, mysqld_multi.sh) use special
option groups with names like [mysqld1], [mysqld2], ..., [mysqldN].

But SST scripts can't currently fully support these option groups.
The only option group-related value it gets from the server is
--defaults-group-suffix, if that option was set for mysqld when
the server was started.

However, the SST scripts does not get told by the server to read
these option groups, so this means that the SST script will fail
to read options like innodb-data-home-dir when it is in a option
group like [mysqld1]...[mysqldN].

Moreover, SST scripts ignore many parameters that can be passed
to them explicitly and cannot transfer them further, for example,
to the input of mariabackup utility. Ideally, we want to transfer
all the parameters of the original mysqld call to utilities such
as mariabackup, however the SST script does not receive these
parameters from the server and therefore cannot transfer them to
mariabackup.

To correct these shortcomings, we need to transfer to the scripts
all of the parameters of the original mysqld call, and in the SST
scripts themselves provide for the transfer all of these parameters
to utilities such as mariabackup. To prevent these parameters from
mixing with the script's own parameters, they should be transferred
to SST script after the special option "--mysqld-args", followed by
the string argument with the original parameters, as it received by
the mysqld call at the time of launch (further all these parameters
will be passed to mariabackup, for example).

In addition, the SST scripts themselves must be refined so that
they can read the parameters from the user-selected group, not just
from the global mysqld configuration group. And also so that they
can receive the parameters (which important for their work) as
command-line arguments.

MemorySanitizer is a compile-time instrumentation layer in clang and GCC.
Together with AddressSanitizer mostly makes the run-time instrumentation
of Valgrind redundant. It is a little more tricky to set up, because
running with uninstrumented libraries will lead into false positives.

You will need an instrumented libc++, and you should use
-stdlib=libc++ instead of the default libstdc++. To build the
instrumented library, you can refer to
https://github.com/google/sanitizers/wiki/MemorySanitizerLibcxxHowTo
or you can adapt these steps that worked for me, for clang-8 version 8.0.1:

cd /mariadb
sudo apt source libc++-8-dev
cd llvm-toolchain-8-8.0.1
mkdir libc++msan; cd libc++msan
cmake ../libcxx -DCMAKE_BUILD_TYPE=Release -DLLVM_USE_SANITIZER=Memory \
-DCMAKE_C_COMPILER=clang-8 -DCMAKE_CXX_COMPILER=clang++-8

Then, in your MariaDB build directory, you have to compile with
libc++ and bundled libraries, such as WITH_SSL=bundled, WITH_ZLIB=bundled.
For uninstrumented system libraries, you will get false positives for
uninitialized values. Like this:

cmake -DWITH_MSAN=ON -DWITH_SSL=bundled -DWITH_ZLIB=bundled \
-DCMAKE_CXX_FLAGS='-stdlib=libc++' ..

Note: you should also add -O2 to the compiler options, or you may
get crashes due to stack overflow.

Finally, to run tests, you must replace libc++ with the instrumented one:

LD_LIBRARY_PATH=/mariadb/llvm-toolchain-8-8.0.1/libc++msan/lib \
MSAN_OPTIONS=abort_on_error=1 \
./mtr --big-test --parallel=auto --force --retry=0

Failure to do so will report numerous false positives related to
operations on std::string and the like.

This is work in progress. Some issues will still have to be fixed
for WITH_MSAN to be usable. See MDEV-20377 for details.

========
During ibd file creation, InnoDB flushes the page0 without crypt
information. During recovery, InnoDB encounters encrypted page read
before initialising the crypt data of the tablespace. So it leads t
corruption of page and doesn't allow innodb to start.

Solution:
=========
Write crypt_data information in page0 while creating .ibd file creation.
During recovery, crypt_data will be initialised while processing
MLOG_FILE_NAME redo log record.

Optimize the test by dropping the table early and by using only
one undo log thread, so that purge will be doing more useful work
and less busy work of suspending and resuming the worker threads.

The test used to cause shutdown timeout on 10.4 on buildbot, and
for me locally when using --mysqld=--innodb-sync-debug.
With these tweaks, it passes for me with --mysqld=--innodb-sync-debug.

It looks like the merge of MySQL 5.7.9 to MariaDB 10.2.2 conflicted with
earlier changes that were made in MDEV-8588.

row_search_mvcc(): If the page is corrupted, avoid invoking
btr_cur_store_position(). The caller should not try to fetch
the next record after a hard error.

This triggered a "may be uninitialized" warning from GCC 9.2.1.
The bogus-looking condition was added in
7e916bb8

Problem:
=======
DROP TABLE IF EXISTS was killed. The table still exists on
the master but the DDL was still logged.

Analysis:
=========
During the execution of DROP TABLE command "ha_delete_table" call is invoked
to delete the table. If the query is killed at this point, the kill command
is not handled within the code. This results in two issues.
1) The table which is not dropped also gets written into the binary log.
2) The code continues further upon receiving 'KILL QUERY'.

Fix:
===
Upon receiving the KILL command the query should stop its current execution.
Tables which were successfully dropped prior to KILL command should be
included in the binary log.

If there're multiple row versions in InnoDB, reading one row from PK
may have O(N) complexity and reading from secondary keys may have
O(N^2) complexity.

The problem occurs when there are many pending versions of the same
row, meaning that the primary key is the same, but a secondary key is
different.  The slowdown occurs when the secondary index is
traversed. This patch creates a helper class for the function
row_sel_get_clust_rec_for_mysql() which can remember and re-use
cached_clust_rec & cached_old_vers so that rec_get_offsets() does not
need to be called over and over for the clustered record.

Corrections by Kevin Lewis <kevin.lewis@oracle.com>

MDEV-20341 Unstable innodb.innodb_bug14704286

Removed test that tested the ability of interrupting long query which
is not long anymore.

Enable the rocksdb test suite. It now passes the valgrind tests.

- Include the valgrind suppressions from the FB upstream

- Use HAVE_Valgrind, not HAVE_Purify (like the rest of MariaDB code does)
  The call to DisownData() is now actually disabled under Valgrind

Use DEBUG_SYNC to hang the execution at the interesting point,
and then kill and restart the server externally. This will work
also with Valgrind. DBUG_SUICIDE() causes Valgrind to hang,
and it could also cause uninteresting reports about memory leaks.

While we are at it, let us clean up innodb.innodb_bulk_create_index_debug
so that it will actually test the desired functionality also in future
versions (with instant ADD COLUMN and DROP COLUMN) and avoid
some unnecessary restarts.

We are adding two DEBUG_SYNC points for ALTER TABLE, because there were
none that would be executed right before ha_commit_trans().

Galera threads were not registered to performance schema and
used pthread_create when mysql_thread_create should have been
used.

Added test case to verify current galera performance schema
instrumentation does work.

Test changes only.

Skip the test on big-endian systems.
In MariaDB Server 10.0 and 10.1 (as well as MySQL 5.6),
the implementation of innodb_checksum_algorithm=crc32
wrongly assumes little-endian byte order.

Fix incorrect else that should have been else if.

MDEV-17614 flags INSERT…ON DUPLICATE KEY UPDATE unsafe for statement-based
replication when there are multiple unique indexes. This correctly fixes
something whose attempted fix in MySQL 5.7
in mysql/mysql-server@c93b0d9a972cb6f98fd445f2b69d924350f9128a
caused lock conflicts. That change was reverted in MySQL 5.7.26
in mysql/mysql-server@066b6fdd433aa6673622341f1a2f0a3a20018043
(with a substantial amount of other changes).

In MDEV-17073 we already disabled the unfortunate MySQL change when
statement-based replication was not being used. Now, thanks to MDEV-17614,
we can actually remove the change altogether.

This reverts commit 8a346f31 (MDEV-17073)
and mysql/mysql-server@c93b0d9a972cb6f98fd445f2b69d924350f9128a while
keeping the test cases.

- mysqltest didn't free read_command_buf
- wait_for_slave_param did write different things to the log if valgrind
  was used.
- Table open cache should not write the initial variable value as it
  can depend on the configuration or if valgrind is used
- A variable in GetResult was used uninitalized

... produces "bytes lost" warnings

When rocksdb_validate_update_cf_options() returns an error,
the update won't happen.
Free the copy of the string in this case.

Problem:-
When mysql executes INSERT ON DUPLICATE KEY INSERT, the storage engine checks
if the inserted row would generate a duplicate key error. If yes, it returns
the existing row to mysql, mysql updates it and sends it back to the storage
engine.When the table has more than one unique or primary key, this statement
is sensitive to the order in which the storage engines checks the keys.
Depending on this order, the storage engine may determine different rows
to mysql, and hence mysql can update different rows.The order that the
storage engine checks keys is not deterministic. For example, InnoDB checks
keys in an order that depends on the order in which indexes were added to
the table. The first added index is checked first. So if master and slave
have added indexes in different orders, then slave may go out of sync.

Solution:-
Make INSERT...ON DUPLICATE KEY UPDATE unsafe while using stmt or mixed format
When there is more then one unique key.
Although there is two exception.
  1. Auto Increment key is not counted because Innodb will get gap lock for
    failed Insert and concurrent insert will get a next increment value. But if
    user supplies auto inc value it can be unsafe.
  2. Count only unique keys for which insertion is performed.

So this patch also addresses the bug id #72921

MDEV-17717
Assertion `!table->pos_in_locked_tables' failed in tc_release_table on
flushing RocksDB table under SERIALIZABLE
MDEV-17998
Deadlock and eventual Assertion `!table->pos_in_locked_tables' failed
in tc_release_table on KILL_TIMEOUT
MDEV-19591
Assertion `!table->pos_in_locked_tables' failed in tc_release_table upon
altering table into S3 under lock.

The problem was that thd->open_tables->pos_in_locked_tables was not reset
when alter table failed to reopen a locked table.

- pcretest.c could use macro with side effect
- maria_chk could access freed memory
- Initialized some variables that could be accessed uninitalized
- Fixed compiler warning in my_atomic-t.c

The general reason why innodb redo log file is limited by 512G is that
log_block_convert_lsn_to_no() returns value limited by 1G. But there is no
need to have unique log block numbers in log group. The fix removes 512G
limit and limits log group size by
(uint32_t maximum value) * (minimum page size), which, in turns, can be
removed if fil_io() is no longer used for innodb redo log io.

- The commit ab6dd774 wrongly sets the
condition inside innobase_srv_conc_enter_innodb().  Problem is that
InnoDB makes the thread to sleep indefinitely if it is a replication
slave thread.

Thanks to Sujatha Sivakumar for contributing the replication test case.