The test was missing a wait_for_binlog_checkpoint.inc, making it non-deterministic
Signed-off-by: Kristian Nielsen <knielsen@knielsen-hq.org>

MDEV-32733: Two JSON related tests running in PS mode fail on server built with -DWITH_PROTECT_STATEMENT_MEMROOT=YES

The tests main.func_json and json.json_no_table fail on server built with
the option -DWITH_PROTECT_STATEMENT_MEMROOT=YES by the reason that a memory
is allocated on the statement's memory root on the second execution of
a query that uses the function json_contains_path().

The reason that a memory is allocated on second execution of a prepared
statement that ivokes the function json_contains_path() is that a memory
allocated on every call of the method Item_json_str_multipath::fix_fields

To fix the issue, memory allocation should be done only once on first
call of the method Item_json_str_multipath::fix_fields. Simmilar issue
take place inside the method Item_func_json_contains_path::fix_fields.
Both methods are modified to make memory allocation only once on its
first execution and later re-use the allocated memory.

Before this patch the memory referenced by the pointers stored in the array
tmp_paths were released by the method Item_func_json_contains_path::cleanup
that is called on finishing execution of a prepared statement. Now that
memory allocation performed inside the method Item_json_str_multipath::fix_fields
is done only once, the item clean up has degenerate form and can be
delegated to the cleanup() method of the base class and memory deallocation
can be performed in the destructor.

The getmntinfo64 interface is deprected in MacOSX12.1.sdk.

Using getmntinfo instead.

Thanks heyingquay for reporting the bug and testing the fix.

This fixes up commit 01031f43

When parsing statements like (SELECT .. FROM ..) ORDER BY <expr>,
there is a step LEX::add_tail_to_query_expression_body_ext_parens()
which calls LEX::wrap_unit_into_derived(). After that the statement
looks like SELECT * FROM (SELECT .. FROM ..), and parser's
Lex_order_limit_lock structure (ORDER BY <expr>) is assigned to
the new SELECT. But what is missing here is that Items in
Lex_order_limit_lock are left with their original name resolution
contexts, and fix_fields() later resolves the names incorrectly.

For example, when processing
  (SELECT * FROM t1 JOIN t2 ON a=b) ORDER BY a
Item_field 'a' in the ORDER BY clause is left with the name resolution
context of the derived table (first_name_resolution_table='t1'), so
it is resolved to 't1.a', which is incorrect.
After LEX::wrap_unit_into_derived() the statement looks like
  SELECT * FROM (SELECT * FROM t1 JOIN t2 ON a=b) AS '__2' ORDER BY a,
and the name resolution context for Item_field 'a' in the ORDER BY
must be set to the wrapping SELECT's one.

This commit fixes the issue by changing context for Items in
Lex_order_limit_lock after LEX::wrap_unit_into_derived().

There are two TABLE objects in each thread: first one is created in
delayed thread by Delayed_insert::open_and_lock_table(), second one is
created in connection thread by Delayed_insert::get_local_table(). It
is copied from the delayed thread table.

When the second table is copied copy-assignment operator copies
vcol_refix_list which is already filled with an item from delayed
thread. Then get_local_table() adds its own item. Thus both tables
contains the same list with two items which is wrong. Then connection
thread finishes and its item freed. Then delayed thread tries to
access it in vcol_cleanup_expr().

The fix just clears vcol_refix_list in the copied table.

Another problem is that copied table contains the same mem_root, any
allocations on it will be invalid if the original table is freed (and
that is indeterministic as it is done in another thread). Since copied
table is allocated in connection THD and lives not longer than
thd->mem_root we may assign its mem_root from thd->mem_root.

Third, it doesn't make sense to do open_and_lock_tables() on NULL
pointer.

mysql_compare_tables() treated all columns non-virtual. Now it
properly checks if the columns are virtual and matches expressions.

When computing vcol expression some items use current_thd and that was
not set in MyISAM repair thread. Since all the repair threads belong
to one connection and items should not write into THD we can utilize
table THD for that.

Attempt to resolve FOR SYSTEM_TIME expression as field for derived
table is done before derived table is fully prepared, so we fail on
assertion that table_list->table is missing.

Actually Vers_history_point::resolve_unit() is done under the call of
mysql_derived_prepare() itself (sql_derived.cc:824) and the table is
assigned later at 867.

The fix disables unit resolution for field type in FOR SYSTEM_TIME
expression as it does a little sense in any case: making historical
queries based on variable field values produces the result of multiple
time points.

fix_fields_if_needed() in resolve_units() was introduced by 46be3198

Index values for row_start/row_end was wrongly calculated for inplace
ALTER for some layout of virtual fields.

Possible impact

  1. history row is not detected upon build clustered index for
     inplace ALTER which may lead to duplicate key errors on
     auto-increment and FTS index add.
  2. foreign key constraint may falsely fail.
  3. after inplace ALTER before server restart trx-based system
     versioning can cause server crash or incorrect data written upon
     UPDATE.

The patch for "MDEV-25440: Indexed CHAR ... broken with NO_PAD collations"
fixed these scenarios from MDEV-26743:
- Basic latin letter vs equal accented letter
- Two letters vs equal (but space padded) expansion

However, this scenario was still broken:
- Basic latin letter (but followed by an ignorable character)
  vs equal accented letter

Fix:
When processing for a NOPAD collation a string with trailing ignorable
characters, like:
  '<non-ignorable><ignorable><ignorable>'

the string gets virtually converted to:
  '<non-ignorable><ignorable><ignorable><space><space><space>...'

After the fix the code works differently in these two cases:
1. <space> fits into the "nchars" limit
2. <space> does not fit into the "nchars" limit

Details:

1. If "nchars" is large enough (4+ in this example),
   return weights as follows:

  '[weight-for-non-ignorable, 1 char] [weight-for-space-character, 3 chars]'

  i.e. the weight for the virtual trailing space character now indicates
  that it corresponds to total 3 characters:
  - two ignorable characters
  - one virtual trailing space character

2. If "nchars" is small (3), then the virtual trailing space character
   does not fit into the "nchar" limit, so return 0x00 as weight, e.g.:

  '[weight-for-non-ignorable, 1 char] [0x00, 2 chars]'

Adding corresponding MTR tests and unit tests.

Problem:

The file backup-my.cnf from the backup directory was loaded by
"mariabackup --prepare" only in case of the explicit --target-dir given.
It was not loaded from the default directory ./xtrabackup_backupfiles/
in case if the explicit --target-dir was missing.

In other words, it worked as follows:

1. When started as "mariabackup --prepare --target-dir=DIR", mariabackup:
  a. loads defaults from "DIR/backup-my.cnf"
  b. processes data files in the specified directory DIR

2. When started as "mariabackup --prepare", mariabackup:
  a. does not load defaults from "./xtrabackup_backupfiles/backup-my.cnf"
  b. processes data files in the default directory "./xtrabackup_backupfiles/"

This patch fixes the second scenario, so it works as follows:

2. When started as "mariabackup --prepare", mariabackup:
  a. loads defaults from "./xtrabackup_backupfiles/backup-my.cnf"
  b. processes data files in the default directory "./xtrabackup_backupfiles/"

This change fixes (among others) the problem with the
"Can't open shared library '/file_key_management.so'" error
reported when "mariabackup --prepare" is used without --target-dir
in combinaton with the encryption plugin.

The crash happened with an indexed virtual column whose
value is evaluated using a function that has a different meaning
in sql_mode='' vs sql_mode=ORACLE:

- DECODE()
- LTRIM()
- RTRIM()
- LPAD()
- RPAD()
- REPLACE()
- SUBSTR()

For example:

CREATE TABLE t1 (
  b VARCHAR(1),
  g CHAR(1) GENERATED ALWAYS AS (SUBSTR(b,0,0)) VIRTUAL,
  KEY g(g)
);

So far we had replacement XXX_ORACLE() functions for all mentioned function,
e.g. SUBSTR_ORACLE() for SUBSTR(). So it was possible to correctly re-parse
SUBSTR_ORACLE() even in sql_mode=''.

But it was not possible to re-parse the MariaDB version of SUBSTR()
after switching to sql_mode=ORACLE. It was erroneously mis-interpreted
as SUBSTR_ORACLE().

As a result, this combination worked fine:

SET sql_mode=ORACLE;
CREATE TABLE t1 ... g CHAR(1) GENERATED ALWAYS AS (SUBSTR(b,0,0)) VIRTUAL, ...;
INSERT ...
FLUSH TABLES;
SET sql_mode='';
INSERT ...

But the other way around it crashed:

SET sql_mode='';
CREATE TABLE t1 ... g CHAR(1) GENERATED ALWAYS AS (SUBSTR(b,0,0)) VIRTUAL, ...;
INSERT ...
FLUSH TABLES;
SET sql_mode=ORACLE;
INSERT ...

At CREATE time, SUBSTR was instantiated as Item_func_substr and printed
in the FRM file as substr(). At re-open time with sql_mode=ORACLE, "substr()"
was erroneously instantiated as Item_func_substr_oracle.

Fix:

The fix proposes a symmetric solution. It provides a way to re-parse reliably
all sql_mode dependent functions to their original CREATE TABLE time meaning,
no matter what the open-time sql_mode is.

We take advantage of the same idea we previously used to resolve sql_mode
dependent data types.

Now all sql_mode dependent functions are printed by SHOW using a schema
qualifier when the current sql_mode differs from the function sql_mode:

SET sql_mode='';
CREATE TABLE t1 ... SUBSTR(a,b,c) ..;
SET sql_mode=ORACLE;
SHOW CREATE TABLE t1;   ->   mariadb_schema.substr(a,b,c)

SET sql_mode=ORACLE;
CREATE TABLE t2 ... SUBSTR(a,b,c) ..;
SET sql_mode='';
SHOW CREATE TABLE t1;   ->   oracle_schema.substr(a,b,c)

Old replacement names like substr_oracle() are still understood for
backward compatibility and used in FRM files (for downgrade compatibility),
but they are not printed by SHOW any more.

This imports and adapts a number of MySQL 5.7 test cases that are
applicable to MariaDB.

Some tests for old bug fixes are not that relevant because the code
has been refactored since then (especially starting with
MariaDB Server 10.6), and the tests would not reproduce the
original bug if the fix was reverted.

In the test innodb_fts.opt, there are many duplicate MATCH ranks, which
would make the results nondeterministic. The test was stabilized by
changing some LIMIT clauses or by adding sorted_result in those cases
where the purpose of a test was to show that no sorting took place
in the server.

In the test innodb_fts.phrase, MySQL 5.7 would generate FTS_DOC_ID that
are 1 larger than in MariaDB. In innodb_fts.index_table the difference is 2.
This is because in MariaDB, fts_get_next_doc_id() post-increments
cache->next_doc_id, while MySQL 5.7 pre-increments it.

Reviewed by: Thirunarayanan Balathandayuthapani

In some cases "SHOW PROCESSLIST" could show "Reset for next command"
as State, even if the previous query had finished properly.

Fixed by clearing State after end of command and also setting the State
for the "Connect" command.

Other things:
- Changed usage of 'thd->set_command(COM_SLEEP)' to
  'thd->mark_connection_idle()'.
- Changed thread_state_info() to return "" instead of NULL. This is
  just a safety measurement and in line with the logic of the
  rest of the function.

Post-review fixes.

necessary functions added to the SQL SERVICE.

ifdef fixed.

test_sql_service.test fixed.

Embedded-server related fixes.

Binary logging is now disabled for the queries run by SQL SERVICE.
The binlogging can be turned on with the 'SET SQL_LOG_BIN=On' query.

Conflicts:
	sql/sql_prepare.cc

Conflicts:
	sql/sql_prepare.cc

   Backported from 10.7.
   The reason for the crash was a bug in MDEV-19275, after which
   shutdown does not wait for binlog threads anymore.

  Fixes to make SQL SERVICE working.

Conflicts:
	storage/spider/spd_table.cc

   The SQL SERVICE backported into the 10.4.

The reason was that Event e11 was re-executed before
"ALTER EVENT e11 DISABLE" had been executed.

Fixed by increasing re-schedule time

Other things:
- Removed double accounting of 'execution_count'. It was incremented in
  top->mark_last_executed(thd) that was executed a few lines earlier.

There where two errors left from the previous fix.

- subselect.test assumes that mysql.slow_log is empty. This was not
  enforced.
- subselect.test dropped a file that does not exists (for safety).
  This was fixed by ensuring we don't get a warning if the file does
  not exist.

Before MariaDB 10.3.5, the binlog position was stored in the TRX_SYS page,
while after it is stored in rollback segments. There is code to read the
legacy position from TRX_SYS to handle upgrades. The problem was if the
legacy position happens to compare larger than the position found in
rollback segments; in this case, the old TRX_SYS position would incorrectly
be preferred over the newer position from rollback segments.

Fixed by always preferring a position from rollback segments over a legacy
position.
Signed-off-by: Kristian Nielsen <knielsen@knielsen-hq.org>

This commit can cause the wrong (old) binlog position to be recovered by
mariabackup --prepare. It implements that the value of the FIL_PAGE_LSN is
compared to determine which binlog position is the last one and should be
recoved. However, it is not guaranteed that the FIL_PAGE_LSN order matches the
commit order, as is assumed by the code. This is because the page LSN could be
modified by an unrelated update of the page after the commit.

In one example, the recovery first encountered this in trx_rseg_mem_restore():

  lsn=27282754  binlog position (./master-bin.000001, 472908)

and then later:

  lsn=27282699  binlog position (./master-bin.000001, 477164)

The last one 477164 is the correct position. However, because the LSN
encountered for the first one is higher, that position is recovered instead.
This results in too old binlog position, and a newly provisioned slave will
start replicating too early and get duplicate key error or similar.
Signed-off-by: Kristian Nielsen <knielsen@knielsen-hq.org>

Signed-off-by: Kristian Nielsen <knielsen@knielsen-hq.org>

Revert the patch for MDEV-18917, which removed this functionality.
This restores that mariabackup --prepare recovers the transactional
binlog position from the redo log, and writes it to the file
xtrabackup_binlog_pos_innodb.

This position is updated only on every InnoDB commit. This means that
if the last event in the binlog at the time of backup is a DDL or
non-transactional update, the recovered position from --no-lock will
be behind the state of the backup.
Signed-off-by: Kristian Nielsen <knielsen@knielsen-hq.org>

list elements not correctly allocated in push_back.

Deprecation versions taken from https://mariadb.com/kb/en/mariabackup-options/#-compress

MDEV-25329: Assertion `allocated_status_memory != __null' failed in void PROF_MEASUREMENT::set_label(const char*, const char*, const char*, unsigned int)

Make profiler to play a bit better with memory allocators. Test suite can not be included because it lead to non free memory on shutdown (IMHO OK for memory shortage emulation)

As alternetive all this should be rewritten and ability to return errors on upper level should be added.