PROBLEM
=======
When add of virtual index fails with DB_TOO_BIG_RECORD , the virtual
index being freed isn't removed from the list of indexes a virtual
column(which is part of the index). This while the undo log is read
could fetch a wrong value during rollback and cause the assertion
reported in the bug particularly.

FIX
===
Added a function that is called when the virtual index being freed would
allow the index be removed from the index list of virtual column which
was a field of that index.

Reviwed By: Jimmy Yang<Jimmy.Yang@oracle.com>
RB: 18528

PROBLEM
-------

Whenever an fts table is created it registers itself in a queue which
is operated by a background thread whose job is to optimize the
fts tables in background. Additionally we place these fts tables in
non-LRU list so that they cannot be evicted from cache. But in the
scenario when a node is brought up which is already having fts
tables ,we first try to load the fts tables in dictionary ,but we skip
the part where it is added in background queue and in non-LRU list because
the background thread is not yet created,so these tables are loaded
but they can be evicted from the cache. Now coming to the deadlock scenario

1. A Server background thread is trying to evict a table from the cache
    because the cache is full,so it scans the LRU list for the tables it can
    evict.It finds the fts table (because of the reason explained above)
    can be evicted and it takes the dict_sys->mutex (this is a system wide mutex)
    submits a request to  the background thread to remove this table from queue
    and waits it to be completed.

2.  In the mean time fts_optimize_thread() is processing another job
    in the queue and needs dict_sys->mutex for a small amount of time,
    but it cannot get it because it is blocked by the first background thread.

So Thread 1 is waiting for its job to be completed by Thread 2,whereas Thread 2
is waiting for dict_sys->mutex held by thread 1 ,causing the deadlock.

FIX

Problem:

when incorrect value is assigned to innodb_data_file_path or
innodb_temp_data_file_path parameter, Innodb returns error and logs error
message in mysqlds.err file but there is no information in error message about
the parameter which causes Innodb initialization is failed.

Fix:

Added error message with parameter name and value, which causes Innodb
initialization is failed.

Reviewed by: Jimmy <Jimmy.Yang@oracle.com>
RB: 18206

Note: it does not appear to cover the bug fix! The test
will pass even if the Oracle Bug #26731689 fix is reverted.

Problem:

During ALTER, when filling stored column info, wrong column number is used.
This is because we ignored virtual column when iterating over columns in
table and lead to debug assertion.

Fix:

In InnoDB table cache object, vcols are on stored on one list, stored and
normal columns are stored in another list.
When looking for stored column, ignore the virtual columns to get the right
column number of stored column.

Reviewed by: Thiru <thirunarayanan.balathandayuth@oracle.com>,
             Satya <satya.bodapati@oracle.com>
RB: 17939

This is the MariaDB equivalent of fixing the MySQL 5.7 regression
Bug #26935001 ALTER TABLE AUTO_INCREMENT TRIES TO READ
INDEX FROM DISCARDED TABLESPACE

Oracle did not publish a test case, but it is easy to guess
based on the commit message. The MariaDB code is different
due to MDEV-6076 implementing persistent AUTO_INCREMENT.

commit_set_autoinc(): Report ER_TABLESPACE_DISCARDED if the
tablespace is missing.

prepare_inplace_alter_table_dict(): Avoid accessing a discarded
tablespace. (This avoids generating warnings in fil_space_acquire().)

Problem:
When FTS index is added into a table which doesn't have 'FTS_DOC_ID'
column, Innodb rebuilds table to add column 'FTS_DOC_ID'. when this FTS
index is dropped from this table. Innodb doesn't not rebuild table to
remove 'FTS_DOC_ID' column and deletes FTS index auxiliary tables.
But it doesn't delete FTS common auxiliary tables.
Later when the database having this table is renamed, FTS auxiliary
tables are not renamed because table's flags2 (dict_table_t.flags2)
has been resetted for DICT_TF2_FTS flag during FTS index drop operation.
Now when we drop old database, it leads to an assert.

Fix:
During renaming of FTS auxiliary tables, ORed a condition to check if
table has DICT_TF2_FTS_HAS_DOC_ID flag set.

RB: 18769
Reviewed by : Jimmy.Yang@oracle.com

Problem:
=======
Multiple insert statement in table contains FULLTEXT KEY and a
FTS_DOC_ID column aborts the server if the FTS_DOC_ID exceeds
FTS_DOC_ID_MAX_STEP.

Solution:
========
Remove the exception for first committed insert statement.

Reviewed-by: Jimmy Yang<jimmy.yang@oracle.com>
RB: 18023

When Oracle fixed MDEV-13899 in their own way, they moved the
condition to the only caller of PageConverter::update_records().
Thus, the merge of 5.6.40 into MariaDB added a redundant condition.

PageConverter::update_records(): Move the page_is_leaf() condition
to the only caller, PageConverter::update_index_page().

Enable the test as the cause of the failure has been fixed

The issue here is that the window function execution is not called for the correct join tab, when we have GROUP BY
where we create extra temporary tables then we need to call window function execution for the last join tab. For doing
so the current code does not take into account the JOIN::aggr_tables.
Fixed by introducing a new function JOIN::total_join_tab_cnt that takes in account the temporary tables also.

It produced warnings due to perl code printing \r\n into the text
file which is then used by LOAD DATA INFILE.

Table name may be passed either as "./db/table" or as ".\\db\\table".

QUERY_DML_NO_SELECT flag added.

QUERY_DML_NO_SELECT flag added.

QUERY_DML_NO_SELECT flag added.

MDEV-15746 ASAN heap-use-after-free in Item_change_list::rollback_item_tree_changes on ALTER executed as PS

don't try to convert a default value string from a user character set
into a column character set, if this particular default value string did
not came from the user at all (that is, if it's an ALTER TABLE and the
default value string is the *old* default value of the unaltered
column).

This used to crash, because old defaults are allocated on the old
table's memroot, which is freed mid-ALTER when the old table is closed.
So thd->rollback_item_tree_changes() at the end of the ALTER was writing
into the freed memory.

table.cc:
  virtual columns must be computed for INSERT, if they're part
  of the partitioning expression.

this change broke gcol.gcol_partition_innodb.
fix CHECK TABLE for partitioned tables and vcols.

sql_partition.cc:
  mark prerequisite base columns in full_part_field_set
ha_partition.cc
  initialize vcol_set accordingly

This previously unreported warning comes from casting size_t to ulong
in sql_hset.h in Hash_Set::at().

Change my_hash_element to accept size_t index parameter.

The remote users need the SUPER privilege because by default Spider sends a
'SET SQL_LOG_OFF' statement to the data nodes.  This is controlled by the
spider_internal_sql_log_off configuration setting on the Spider node, which
can only be set to 0 or 1, with a default value of 1.

I have fixed the problem by changing this configuration setting so that if it
is NOT SET, which is the most likely case, the Spider node DOES NOT SEND the
'SET SQL_LOG_OFF' statement to the data nodes.  However if the
spider_internal_sql_log_off setting IS EXPLICITLY SET to either 0 or 1, then
the Spider node DOES SEND the 'SET SQL_LOG_OFF' statement, requiring a remote
user with the SUPER privilege.  The Spider documentation will be updated to
reflect this change.

Author:
  Jacob Mathew.

Reviewer:
  Kentoku Shiba.

Cherry-Picked:
  Commit 72f0efac on branch bb-10.3-MDEV-15697

Problem:
=======
InnoDB master thread encounters the shutdown state as SRV_SHUTDOWN_FLUSH_PHASE
when innodb_force_recovery >=2 and slow scheduling of master thread during
shutdown.

Fix:
====
There is no need for master thread itself if innodb_force_recovery >=2.
Don't create the master thread if innodb_force_recovery >= 2

The problem is hard to repeat, and I failed to create a deterministic
test case. Online index creation creates stubs for to-be-created indexes.
If index creation fails, we could remove these stubs while locks exist
in the indexes. (This would require that the index creation was completed,
and a concurrent DML operation acquired a lock on a record in the
uncommitted index. If a duplicate key error occurs in an uncommitted
index, the error will be reported for the CREATE UNIQUE INDEX, not for
the DML operation that tried to insert the duplicate.)

dict_table_try_drop_aborted(), row_merge_drop_indexes(): If transactional
locks exist on the table, keep the table->indexes intact.

ibuf_restore_pos(): Do not issue any messages if the tablespace
is being dropped or truncated. In MariaDB 10.2, TRUNCATE TABLE
would not change the tablespace ID, and therefore the tablespace
would be found, even though TRUNCATE is in progress.

Furthermore, do not commit suicide if restoring the change buffer
cursor fails. The worst that could happen is that a secondary index
becomes corrupted due to incomplete change buffer merge.

Remove the setup_ports function call. This is related to
https://github.com/MariaDB/server/pull/717

Thanks to Daniel Black and Bart S.

The reason is the missing HAVE_OPENSSL define for mariabackup.

The following variables are used in this project, but they are set to NOTFOUND.
LZ4_LIBS

The reason for the failure is that pkg_check_modules will not guarantee
<prefix>_LIBRARY_DIRS variable to be set, according to documentation.
When it's not set, we would force find_library to look in an empty path
and thus fail to correctly find LZ4_LIBS, although pck_check_modules
did previously discover that the library is installed.

To fix the problem and still keep the logic of first following
LIBLZ4_LIBRARY_DIRS and *then* look at other paths, we call find_library
twice. This is the recommended approach, according to CMake 3.11
documentation.