The function btr_pcur_close() is being invoked on local variables
even when no cleanup needs to be done. In particular, for B-tree
indexes (not SPATIAL INDEX), unless btr_pcur_store_position()
was invoked in the past, there is no need to invoke btr_pcur_close().

On purge and rollback, we will retain btr_pcur_close(&pcur)
because otherwise some ./mtr --suite=innodb_gis tests would leak memory.

Since commit 2ca11234 (MDEV-26217)
the function trx_t::commit(std::vector<pfs_os_file_t>&)
holds exclusive lock_sys.latch while invoking fil_delete_tablespace(),
which in turn may wait for change buffer tree latches in
ibuf_delete_for_discarded_space().

ibuf_insert_low(): If a shared lock_sys.latch cannot be acquired
without waiting, refuse to buffer the insert. In this way, a
deadlock with a DDL operation will be avoided.

ibuf_insert_to_index_page(), ibuf_delete(): Remove redundant calls to
record locking. In ibuf_insert_low() we already ensured that no record
locks existed on the page. No locks can be added before the buffered
changes have been merged.

- Alter takes index lock on commit phase twice while applying the
online log for the index build. This leads to assert failure and caused
by MDEV-15250 (4b80c11f).

Problem:
========
 InnoDB DDL fails to remove the newly added table or index from
dictionary and index stub from the table cache if the alter
transaction encounters DEADLOCK error in commit phase.

Solution:
========
Restart the alter table transaction if it encounters DEADLOCK
in commit phase. So that index stubs and index, table removal from
dictionary can be done in rollback_inplace_alter_table()

- Added one assert in rollback_inplace_alter_table() to indicate that
the online log for the old table shouldn't exist.

Two submodules were inadvertently reverted to an older revision
in commit 2ca11234.
Thanks to Roel Van de Paar for noticing this.

lock_sec_rec_read_check_and_lock(): Remove a redundant check
for trx_sys.get_min_trx_id(). This would be checked in
lock_sec_rec_some_has_impl() anyway. Also, test the
cheapest conditions first.

lock_sec_rec_some_has_impl(): Replace trx_sys.get_min_trx_id()
with trx_sys.find_same_or_older() that is much easier to evaluate.

Inspired by mysql/mysql-server@0a0c0db97e9c9705f4dffc629a770fb87a60cb22

Problem:
=======
Index page latches must be acquired before undo page latches.
In trx_t::apply_log(), InnoDB acquired undo log page latch
before an index page latch.

Solution:
========
In trx_t::apply_log(), InnoDB should copy the undo log
record and release the undo log block before applying
it on online indexes.

In commit 1bd681c8 (MDEV-25506 part 3)
the way how DDL transactions delete files was rewritten.
Only files that are actually attached to InnoDB tablespaces would be
deleted, and only after the DDL transaction was durably committed.

After a failed ALTER TABLE...IMPORT TABLESPACE, any data files that
the user might have moved to the data directory will not be attached
to the InnoDB data dictionary. Therefore, DROP TABLE would not
attempt to delete those files, and a subsequent CREATE TABLE would
fail. The logic was that the user who created the files outside the
DBMS is still the owner of those files, and InnoDB should not delete
those files because an "ownership transfer" (IMPORT TABLESPACE) was
not successfully completed.

However, not deleting those detached files could surprise users.

ha_innobase::delete_table(): Even if no tablespace exists, try to
delete any files that might match the table name.

Reviewed by: Thirunarayanan Balathandayuthapani

Recreate mysql_sys_schema.sql, whenever one of its source files changes.

Also the ENOSYS is more likely explained by seccomp
filters in containers than a pre-5.1 kernel, so include
both.

dict_stats_save(): Do not attempt to commit an already committed
transaction.

MDEV-26217 Failing assertion: list.count > 0 in ut_list_remove or Assertion `lock->trx == this' failed in dberr_t trx_t::drop_table

This follows up the previous fix in
commit c3c53926 (MDEV-26554).

ha_innobase::delete_table(): Work around the insufficient
metadata locking (MDL) during DML operations by acquiring exclusive
InnoDB table locks on all child tables. Previously, this was only
done on TRUNCATE and ALTER.

ibuf_delete_rec(), btr_cur_optimistic_delete(): Do not invoke
lock_update_delete() during change buffer operations.
The revised trx_t::commit(std::vector<pfs_os_file_t>&) will
hold exclusive lock_sys.latch while invoking fil_delete_tablespace(),
which in turn may invoke ibuf_delete_rec().

dict_index_t::has_locking(): A new predicate, replacing the dummy
!dict_table_is_locking_disabled(index->table). Used for skipping lock
operations during ibuf_delete_rec().

trx_t::commit(std::vector<pfs_os_file_t>&): Release the locks
and remove the table from the cache while holding exclusive
lock_sys.latch.

trx_t::commit_in_memory(): Skip release_locks() if dict_operation holds.

trx_t::commit(): Reset dict_operation before invoking commit_in_memory()
via commit_persist().

lock_release_on_drop(): Release locks while lock_sys.latch is
exclusively locked.

lock_table(): Add a parameter for a pointer to the table.
We must not dereference the table before a lock_sys.latch has
been acquired. If the pointer to the table does not match the table
at that point, the table is invalid and DB_DEADLOCK will be returned.

row_ins_foreign_check_on_constraint(): Improve the checks.
Remove a bogus DB_LOCK_WAIT_TIMEOUT return that was needed
before commit c5fd9aa5 (MDEV-25919).

row_upd_check_references_constraints(),
wsrep_row_upd_check_foreign_constraints(): Simplify checks.

The macro MY_GNUC_PREREQ() was used for testing for some minor
GCC 4 versions before GCC 4.8.5, which is the oldest version
that supports C++11, which we depend on ever since
commit d9613b75

- InnoDB should avoid bulk insert operation when table has active
DDL. Because bulk insert writes only one undo log as TRX_UNDO_EMPTY
and logging of concurrent DML happens at commit time uses undo log
record to parse and get the value and operation.

- Removed ROW_T_EMPTY, ROW_OP_EMPTY and their associated functions
and also the test case which tries to log the ROW_OP_EMPTY
when table has active DDL.

This commit fixes a crash reported as MDEV-28377 and a number
of other crashes in automated tests with mtr that are related
to broken .cnf files in galera and galera_3nodes suites, which
happened when automatically migrating MDEV-26171 from 10.3 to
subsequent higher versions.

- InnoDB DDL results in `Duplicate entry' if concurrent DML throws
duplicate key error. The following scenario explains the problem

connection con1:
  ALTER TABLE t1 FORCE;

connection con2:
  INSERT INTO t1(pk, uk) VALUES (2, 2), (3, 2);

In connection con2, InnoDB throws the 'DUPLICATE KEY' error because
of unique index. Alter operation will throw the error when applying
the concurrent DML log.

- Inserting the duplicate key for unique index logs the insert
operation for online ALTER TABLE. When insertion fails,
transaction does rollback and it leads to logging of
delete operation for online ALTER TABLE.
While applying the insert log entries, alter operation
encounters 'DUPLICATE KEY' error.

- To avoid the above fake duplicate scenario, InnoDB should
not write any log for online ALTER TABLE before DML transaction
commit.

- User thread which does DML can apply the online log if
InnoDB ran out of online log and index is marked as completed.
Set online log error if apply phase encountered any error.
It can also clear all other indexes log, marks the newly
added indexes as corrupted.

- Removed the old online code which was a part of DML operations

commit_inplace_alter_table() : Does apply the online log
for the last batch of secondary index log and does frees
the log for the completed index.

trx_t::apply_online_log: Set to true while writing the undo
log if the modified table has active DDL

trx_t::apply_log(): Apply the DML changes to online DDL tables

dict_table_t::is_active_ddl(): Returns true if the table
has an active DDL

dict_index_t::online_log_make_dummy(): Assign dummy value
for clustered index online log to indicate the secondary
indexes are being rebuild.

dict_index_t::online_log_is_dummy(): Check whether the online
log has dummy value

ha_innobase_inplace_ctx::log_failure(): Handle the apply log
failure for online DDL transaction

row_log_mark_other_online_index_abort(): Clear out all other
online index log after encountering the error during
row_log_apply()

row_log_get_error(): Get the error happened during row_log_apply()

row_log_online_op(): Does apply the online log if index is
completed and ran out of memory. Returns false if apply log fails

UndorecApplier: Introduced a class to maintain the undo log
record, latched undo buffer page, parse the undo log record,
maintain the undo record type, info bits and update vector

UndorecApplier::get_old_rec(): Get the correct version of the
clustered index record that was modified by the current undo
log record

UndorecApplier::clear_undo_rec(): Clear the undo log related
information after applying the undo log record

UndorecApplier::log_update(): Handle the update, delete undo
log and apply it on online indexes

UndorecApplier::log_insert(): Handle the insert undo log
and apply it on online indexes

UndorecApplier::is_same(): Check whether the given roll pointer
is generated by the current undo log record information

trx_t::rollback_low(): Set apply_online_log for the transaction
after partially rollbacked transaction has any active DDL

prepare_inplace_alter_table_dict(): After allocating the online
log, InnoDB does create fulltext common tables. Fulltext index
doesn't allow the index to be online. So removed the dead
code of online log removal

Thanks to Marko Mäkelä for providing the initial prototype and
Matthias Leich for testing the issue patiently.

These were replaced with innodb.innodb_page_compressed
in commit 35095c45

It suffices to test compression with one record. Restarting the
server is not really needed; we are exercising the log based recovery
in other tests, such as mariabackup.page_compression_level.

The purpose of the compress() wrapper my_compress_buffer() was twofold:
silence Valgrind warnings about uninitialized memory access before
zlib 1.2.4, and have PERFORMANCE_SCHEMA instrumentation of some zlib
related memory allocation. Because of PERFORMANCE_SCHEMA, we cannot
trivially replace my_compress_buffer() with compress().

az_open(): Remove a crc32() call. Any CRC of the empty string is 0.

The option --symbolic-links was originally disabled by default under
Purify (and later Valgrind) in 51156c5a
without any explanation.

Adding a 10.6 specific test from the MDEV

This is a backport of commit 4489a89c
in order to remove the test innodb.redo_log_during_checkpoint
that would cause trouble in the DBUG subsystem invoked by
safe_mutex_lock() via log_checkpoint(). Before
commit 7cffb5f6
these mutexes were of different type.

The following options were introduced in
commit 2e814d47 (mariadb-10.2.2)
and have little use:

innodb_disable_resize_buffer_pool_debug had no effect even in
MariaDB 10.2.2 or MySQL 5.7.9. It was introduced in
mysql/mysql-server@5c4094cf4971eebab89da4ee4ae92c71f69cd524
to work around a problem that was fixed in
mysql/mysql-server@2957ae4f990bf3aed25822b0ce15d3ccad0b54b6
(but the parameter was not removed).

innodb_page_cleaner_disabled_debug and innodb_master_thread_disabled_debug
are only used by the test innodb.redo_log_during_checkpoint
that will be removed as part of this commit.

innodb_dict_stats_disabled_debug is only used by that test,
and it is redundant because one could simply use
innodb_stats_persistent=OFF or the STATS_PERSISTENT=0 attribute
of the table in the test to achieve the same effect.

MDEV-27919 mariabackup --log-copy-interval is measured in millisecondss in 10.5 and in microseconds in 10.6

Multiply polling interval by 1000.

Due to 32-bit arithmetics, SRV_TMP_SPACE_ID page number 0x200002 would be
folded to 0, which is incompatible with the assumption that was made in
commit 7cffb5f6 (MDEV-23399).

page_id_t::fold(): Compute in the native word width instead of uint32_t.
On 64-bit platforms, an alternative would be to return the 64-bit m_id
directly, but that was measured to cause a performance regression.

fil_space_t::open(): Invoke fil_node_t::find_metadata() when the
tablespace is being created. In this way, we will actually detect
that the temporary tablespace resides on SSD. (During database
creation, also the system tablespace will correctly be detected as
residing on SSD.)

I missed include/no_protocol.inc already existed.

aarch64 RHEL7 and Centos7 internal compiler error here, so we carry
another pragma enchantment like buf0lru.cc and row0ins.cc.

The only purpose of ibuf_bitmap_mutex is to prevent a deadlock between
two concurrent invocations of ibuf_update_free_bits_for_two_pages_low()
on the same pair of bitmap pages, but in opposite order.
The mutex is unnecessarily serializing the execution of the function
even when it is being invoked on totally different tablespaces.
To avoid deadlocks, it suffices to ensure that the two page latches
are being acquired in a deterministic (sorted) order.