As initially most tests fail, they have allow_failures defined so that
testing anyway proceeds all the way to the final 'upgrade extras' stage.

All of these tests work for downstream Debian packaging of MariaDB 10.4
and should eventually pass on upstream MariaDB 10.5 as well.

Also upstream the Debian autopkgtests from MariaDB 10.4 in Debian so that
pipeline includes running mtr.

This are filed as issues so I trust these will not be forgotten:
- MDEV-20372
- MDEV-21965
- MDEV-21968
- MDEV-22119

Having these tests fail the MTR permanently decreases the overall quality
of the software as:
- developers start neglecting the results from CI systems and do not spot
  the new regressions their changes introduce
- reviews take many hours as a diligent reviewer needs to open all build
  logs that are failing, see what is failing, and check if those are new
  failures or not – alternatively reviewers stop looking at CI altogether

It is better to disable these test now and prioritize the MDEVs for the
test failure bugs than keep the MTR red and allow more test failures to
creep in.

Test do we have galera debug library with debug_sync functionality
needs to be earlier.

Also add a check for tmp file being empty and bail out with a clear
error message in such a case, as mysqld_safe prevents normal stderr
from being displayed anywhere and would fail silently on this.

MDEV-22181 : galera.galera_sst_mysqldump_with_key MTR failed: INSERT failed: 1146: Table 'test.t1' doesn't exist

Add wait condition to make sure table test.t1 is replicated
to node_2 before we insert to it.

Commit 5defdc38
(which aimed to reduce sizeof(mtr_t) for non-debug builds)
replaced the ternary mtr_t::status with two debug-only bool
data members m_start, m_commit and inadvertently made the
(now debug-only) predicate mtr_t::is_active() wrongly hold
after mtr_t::commit().

mtr_t::is_active(): Evaluate both m_start and m_commit,
to be compatible with the old definition.

row_merge_read_clustered_index(): Correct a debug assertion.

Fix a test result.

commit 105b879d introduced this
warning. The warning looks harmless, but GCC does not understand
that the initialization and the use of the variables are guarded
by the same predicate.

The test was broken in commit f40ca33b.
The background DROP TABLE queue in InnoDB will continue to
use names like #sql-ib, and we must filter out those file names.

When commit 562c037b
removed FIL_TYPE_LOG, the function fil_type_is_data()
became redundant, that is, always returning true for
a valid value of fil_space_t::purpose. Remove it.

f9f33b85 introduced a potential use of
unitialized value when the if condition is false.

Create symlinks during configure time and install them. This is
necessary as Alias support from systemd service file was dropped with:
6af0bd69

* Also ignore the generated symlinks in gitignore

sprintf() format of double changed from '%lg' to '%-.11lg'

The change was to make it easier to read optimizer trace output
with tables that has millions of records.

The reason for this is to make all temporary file names similar and
also to be able to figure out from where a #sql-xxx name orginates.

New format is for most cases:
'#sql-name-current_pid-thread_id[-increment]'
Where name is one of subselect, alter, exchange, temptable or backup

The exceptions are:
ALTER PARTITION shadow files:
'#sql-shadow-thread_id-'original_table_name'

Names used with temp pool:
'#sql-name-current_pid-pool_number'

MDEV-22088 S3 partitioning support

All ALTER PARTITION commands should now work on S3 tables except

REBUILD PARTITION
TRUNCATE PARTITION
REORGANIZE PARTITION

In addition, PARTIONED S3 TABLES can also be replicated.
This is achived by storing the partition tables .frm and .par file on S3
for partitioned shared (S3) tables.

The discovery methods are enchanced by allowing engines that supports
discovery to also support of the partitioned tables .frm and .par file

Things in more detail

- The .frm and .par files of partitioned tables are stored in S3 and kept
  in sync.
- Added hton callback create_partitioning_metadata to inform handler
  that metadata for a partitoned file has changed
- Added back handler::discover_check_version() to be able to check if
  a table's or a part table's definition has changed.
- Added handler::check_if_updates_are_ignored(). Needed for partitioning.
- Renamed rebind() -> rebind_psi(), as it was before.
- Changed CHF_xxx hadnler flags to an enum
- Changed some checks from using table->file->ht to use
  table->file->partition_ht() to get discovery to work with partitioning.
- If TABLE_SHARE::init_from_binary_frm_image() fails, ensure that we
  don't leave any .frm or .par files around.
- Fixed that writefrm() doesn't leave unusable .frm files around
- Appended extension to path for writefrm() to be able to reuse to function
  for creating .par files.
- Added DBUG_PUSH("") to a a few functions that caused a lot of not
  critical tracing.

MDEV-22199 Add VISIBLE attribute for indexes in CREATE TABLE

This was done to make it easier to read in dumps from MySQL 8.0 generated
with MySQL workbench

This was done to increase performance when not using temporary tables
as checking if a table is a temporary table involves a lot of code.

Other things:
- Handler errors from ha_maria::implict_commit
- Disable DBUG in safe_mutex_lock to get trace file easier to read

- MDEV-22062 Assertion `!table->file->keyread_enabled()' failed in
  close_thread_table()
- MDEV-22077 table->no_keyread .. failed in join_read_first()
- MDEV-22237 Assertion `!table->file->keyread_enabled()' failed in
  handler::ha_reset on DELETE

              >= M_TOT_PARTS' FAILED.

This patch is taken from MySQL, originally written by Mattias Jonsson
Here follows the original commit message:

Problem in handle_alter_part_error(),
result in altered partition_info object was still used
if table was under LOCK TABLES.

Solution was to always close and destroy all table
and table_share instances if exclusive mdl lock was
possible.
If not succeeding in get an exlusive lock (only possible
during rollback of DDL), at least close and destroy this
table instance.

rb#7361.
Approved by Mikael and Aditya.

With these in place Lintian will pass without errors (only warnings) and
the CI systems will at least be able to stop anybody from introducing
new Lintian errors.

Fix Lintian complaints on:
 P: maintainer-script-without-set-e
 W: mariadb-server-10.5: script-not-executable usr/share/mysql/mysql.init
 W: mariadb-client-10.5: incorrect-path-for-interpreter usr/bin/innotop
    (#!/usr/bin/env perl != /usr/bin/perl)

Manages the security risk in way that also fixes Lintian warning:
  W: mariadb-server-10.5: setuid-binary
  usr/lib/mysql/plugin/auth_pam_tool_dir/auth_pam_tool 4755 root/root

Applied downstream in
https://salsa.debian.org/mariadb-team/mariadb-10.4/-/commit/9605a48a99a61f439698676da192799de45d732a

There are still some differences in due to systemd and service triggers,
but these differences make sense and are likely to be synced the other
way later so that downstream Debian official packaging adopts them.

The synced changes include among others:
- https://salsa.debian.org/mariadb-team/mariadb-10.4/-/commit/9f49e4b494f3dad8c403972996f7a1ebceb4b34f
- https://salsa.debian.org/mariadb-team/mariadb-10.4/-/commit/6440c0d6e75aeda134f4f942eee4fbfd62b1ce20
- https://salsa.debian.org/mariadb-team/mariadb-10.4/-/commit/6e5ee72d647744bf84c42e02c3dfe888912e739e
- https://salsa.debian.org/mariadb-team/mariadb-10.4/-/commit/e62e67ae4b7679c43e80a20ab3ba3c51f01f0714
- https://salsa.debian.org/mariadb-team/mariadb-10.4/-/commit/df2415a53d5604755768f00aad0c640df1ad815c
- https://salsa.debian.org/mariadb-team/mariadb-10.4/-/commit/643558da74990f8648b3c86039e253dbae337c84

These changes are in line with what has been changes in downstream Debian
and they fix the following errors:

 $ check_for_missing_breaks_replaces.py -o ${WORKING_DIR}/missing_breaks.xml --changes-file ${WORKING_DIR}/*.changes
 [ERROR] mariadb-client-10.5 conflicts with mytop files: {'/usr/bin/mytop'}
 [ERROR] mariadb-client-10.5 conflicts with mariadb-client-core-10.3 files: {'/usr/bin/mysqlcheck'}
 [ERROR] mariadb-server-10.5 conflicts with mariadb-server-core-10.3 files: {'/usr/share/man/man1/resolveip.1.gz', '/usr/bin/resolveip', '/usr/share/mysql/mysql_test_data_timezone.sql'}
 [ERROR] mariadb-client-10.5 conflicts with mysql-client-core-5.7 files: {'/usr/bin/mysqlcheck'} [ERROR] mariadb-server-10.5 conflicts with mariadb-server-core-10.3 files: {'/usr/share/man/man1/resolveip.1.gz', '/usr/bin/resolveip', '/usr/share/mysql/mysql_test_data_timezone.sql'}
 [ERROR] mariadb-server-10.5 conflicts with mysql-client-5.7 files: {'/usr/bin/myisam_ftdump', '/usr/share/man/man1/myisam_ftdump.1.gz'}
 [ERROR] mariadb-server-core-10.5 conflicts with mysql-server-5.7 files: {'/usr/share/mysql/mysql_test_data_timezone.sql', '/usr/bin/resolveip', '/usr/share/man/man1/resolveip.1.gz'}
 [ERROR] mariadb-test conflicts with mysql-server-5.7 files: {'/usr/lib/mysql/plugin/adt_null.so', '/usr/lib/mysql/plugin/mypluglib.so'}

- Ensure cmake builds also apply CPPFLAGS flags for hardening to fully work
  https://salsa.debian.org/mariadb-team/mariadb-10.4/-/commit/fc4f33cf40d0a10ef5d1992accd2af734ba96356
- Install Ubuntu Apport files in the same way with same filename.
- MDEV-21705: Build flags to keep WolfSSL from crashing
- Clean away sql-bench from packaging immediately after build step.
- Delete private files from libraries so they don't get shipped in the -dev
  packages.

- Make full contents, also header files, explicit in -dev packages
  https://salsa.debian.org/mariadb-team/mariadb-10.4/-/commit/fb77df0c75422449c32ab1f5f8650a7e9e91091f
- Don't ship private header files at all.
- Ship both libmariadb.pc and maridb.pc.
- Ship mysql_config only as a symlink.
- Keep 'mariadbcheck' as symlink as somebody could have used it already
  https://salsa.debian.org/mariadb-team/mariadb-10.3/commit/d0466f74d543ac574dcd832e081fdc18538f3edd
- The mariadb.pc is the server pkg-config, ship in libmariadbd-dev.
- The client pkg-config is called libmariadb.pc. Ship only it in the client
  library packages.

Use versioned breaks/replaces for mysql* packages we also provide and fix:

  [ERROR] libmariadb-dev conflicts with libmysqlclient-dev files:
    {'/usr/share/man/man1/mysql_config.1.gz', '/usr/share/aclocal/mysql.m4'}
  [ERROR] libmariadb-dev conflicts with libmysqld-dev files:
    {'/usr/lib/x86_64-linux-gnu/libmysqlservices.a'}

- Remove versioned depends that are far outdated and not relevant anymore
  in 2020.
- Link with libedit instead of unmaintained readline5
  https://salsa.debian.org/mariadb-team/mariadb-10.4/-/commit/d8796a0e3b5fc4450f4dc80fd159599a7f03997a
- Add [linux-any] for Linux specific depends and other smaller control
  file tweaks.
- Update .travis-ci.yml to match.
- Also fixes the Debian autopkgtest detected error:
    ERROR 1231 (42000) at line 1: Variable 'innodb_compression_algorithm'
    can't be set to the value of 'lz4'

Drop Aliases from the service file directive. Aliases in the service
file only take effect when the service is enabled. This is the case
because Aliases in service files do not have to be unique across various
services.

Shipping symlinks guarantees that one can always enable mariadb service
using `systemctl enable mysql` or `systemctl enable mysqld` and makes
the commands indempotent.

Post change of 8447edb7, --plugin-dir
is set in the default my.cnf file under [client]. The value is automatically
detected during mtr startup.

MDEV-20502 Queries against spider tables return wrong values for columns following constant declarations.

When executing a query like "select id, 0 as const, val from ...", there are 3 columns(items) in Query->select at handlerton->create_group_by(). After that, MariaDB makes a temporary table with 2 columns. The skipped items are const item, so fixing Spider to skip const items for items at Query->select.

- On Windows, do not handle lack of SeLockMemory privilege as fatal error.
Just like on any other platform, there is a fallback to ordinary pages.
It is better than server that does not start, silently.

- On Windows, remove incorrect  irritating "fallback to conventional pages failed"
from the warning, when allocating large pages fails.

The test innodb.innodb_wl6326 that had been disabled in 10.4 due to
MDEV-21535 is failing on 10.5 due to a different reason: the removal
of the MLOG_COMP_END_COPY_CREATED operations in MDEV-12353
commit 276f996a caused PAGE_LAST_INSERT
to be set to something nonzero by the function page_copy_rec_list_end().

This in turn would cause btr_page_get_split_rec_to_right() to behave
differently: we would not attempt to split the page at all, but simply
insert the new record into the new, empty, right leaf page.

Even though the change reduced the sizes of some tables, it is better
to aim for balanced trees.

page_copy_rec_list_end(), PageBulk::finishPage():
Preserve PAGE_LAST_INSERT, PAGE_N_DIRECTION, PAGE_DIRECTION.

PageBulk::finish(): Move some common code from PageBulk::finishPage().

Without this, the mtr will fail to load client plugins and the following
tests would fail:

- main.plugin_auth_qa_2
- main.plugin_auth_qa_3
- main.plugin_auth
- plugins.multiauth
- plugins.auth_ed25519