Make sure that SELECT_LEX_UNIT::derived, behaves as documented
(points to the "TABLE_LIST representing this union in the
embedding select"). For recursive CTE this was not necessarily
the case, it could've pointed to the TABLE_LIST inside the CTE,
not in the embedding select.

To fix:
* don't update unit->derived in mysql_derived_prepare(), pass derived
  as an argument to st_select_lex_unit::prepare()
* prefer to set unit->derived in TABLE_LIST::init_derived()
  to the TABLE_LIST in the embedding select, not to the recursive
  reference. Fail if there are many TABLE_LISTs in the embedding
  select with conflicting FOR SYSTEM_TIME clauses.

cleanup:
* remove redundant THD* argument from st_select_lex_unit::prepare()

it's internal storage engine error, don't let it leak
into the upper layer.

Make --gdb to take an optional argument *only* if it's
written after '=', not after a space.

followup for 339b9055

Compressed blob columns didn't accept data at their capacity. E.g. storing
255 bytes to TINYBLOB results in "Data too long" error.

Now it is allowed assuming compression method was able to produce shorter
string (so that both metadata and compressed data fits blob) and
column_compression_threshold is lower than blob.

If no compression was performed, we still have to reserve additional byte
for metadata and thus we perform normal data truncation and return it's
status.

1. Adding THD::convert_string(LEX_CSTRING *to,...) as a wrapper
   for convert_string(LEX_STRING *to,...), as LEX_CSTRING
   is now frequently used for conversion purpose.
   This reduced duplicate code in TEXT_STRING_sys,
   TEXT_STRING_literal, TEXT_STRING_filesystem grammar rules in *.yy

2. Adding yet another THD::convert_string() with an extra parameter
   "bool simple_copy_is_possible". This even more reduced
   repeatable code in the mentioned grammar rules in *.yy

3. Deriving Lex_ident_cli_st from Lex_string_with_metadata_st,
   as they have very similar functionality. Moving m_quote
   from Lex_ident_cli_st to Lex_string_with_metadata_st,
   as m_quote will be used later to optimize string literals anyway
   (e.g. avoid redundant copying on the tokenizer stage).
   Adjusting Lex_input_stream::get_text() accordingly.

4. Moving the reminders of the code in TEXT_STRING_sys, TEXT_STRING_literal,
   TEXT_STRING_filesystem grammar rules as new methods in THD:
   - make_text_string_sys()
   - make_text_string_connection()
   - make_text_string_filesystem()
   and changing *.yy to use these new methods.
   This reduced the amount of similar code in
   sql_yacc.yy and sql_yacc_ora.yy.

5. Removing duplicate code in Lex_input_stream::body_utf8_append_ident():
   by reusing THD::make_text_string_sys(). Thanks to #3 and #4.

6. Making THD members charset_is_system_charset,
   charset_is_collation_connection, charset_is_character_set_filesystem
   private, as they are not needed externally any more.

item->val_int() == bool

MDEV-15746 ASAN heap-use-after-free in Item_change_list::rollback_item_tree_changes on ALTER executed as PS

10.3+ fix

On ALTER TABLE, if a non-changed column default
might need a charset conversion, it must
be a blob. Because blob's defaults ar stored
as expressions, and for any other type
a basic_const_item() will be in the record,
so it'll have correct charset and won't need
converting. For the same reason it makes no
sense to convert blob defaults (and it's
unsafe, see MDEV-15746).

test case is already in main/ps.test

The code in the "sp_tail" rule in sql_yacc.yy always
used YYLIP->get_cpp_tok_start() as the start of the body,
and did not check for possible lookahead which happens
for keywords "FOR", "VALUES" and "WITH" for LALR(2)
resolution in Lex_input_stream::lex_token().

In case of the lookahead token presence,
get_tok_start_prev() should have been used instead
of get_cpp_tok_start() as the beginning of the SP body.

Change summary:

This patch hides the implementation of the lookahead
token completely inside Lex_input_stream.
The users of Lex_input_stream now just get token-by-token
transparently and should not care about lookahead any more.
Now external users of Lex_input_stream
are not aware of the lookahead token at all.

Change details:

- Moving Lex_input_stream::has_lookahead() into the "private" section.

- Removing Lex_input_stream::get_tok_start_prev() and
  Lex_input_stream::get_cpp_start_prev().

- Fixing the external code to call get_tok_start() and get_cpp_tok_start()
  in all places where get_tok_start_prev() and get_cpp_start_prev()
  where used.

- Adding a test for has_lookahead() right inside
  get_tok_start() and get_cpp_tok_start().
  If there is a lookahead token, these methods now
  return the position of the previous token automatically:

   const char *get_tok_start()
   {
     return has_lookahead() ? m_tok_start_prev : m_tok_start;
   }

   const char *get_cpp_tok_start()
   {
    return has_lookahead() ? m_cpp_tok_start_prev : m_cpp_tok_start;
   }

- Fixing the internal code inside Lex_input_stream methods
  to use m_tok_start and m_cpp_tok_start directly,
  instead of calling get_tok_start() and get_cpp_tok_start(),
  to make sure to access to the *current* token position
  (independently of a lookahead token presence).

MDEV-15746 ASAN heap-use-after-free in Item_change_list::rollback_item_tree_changes on ALTER executed as PS

don't try to convert a default value string from a user character set
into a column character set, if this particular default value string did
not came from the user at all (that is, if it's an ALTER TABLE and the
default value string is the *old* default value of the unaltered
column).

This used to crash, because old defaults are allocated on the old
table's memroot, which is freed mid-ALTER when the old table is closed.
So thd->rollback_item_tree_changes() at the end of the ALTER was writing
into the freed memory.

table.cc:
  virtual columns must be computed for INSERT, if they're part
  of the partitioning expression.

this change broke gcol.gcol_partition_innodb.
fix CHECK TABLE for partitioned tables and vcols.

sql_partition.cc:
  mark prerequisite base columns in full_part_field_set
ha_partition.cc
  initialize vcol_set accordingly

This previously unreported warning comes from casting size_t to ulong
in sql_hset.h in Hash_Set::at().

Change my_hash_element to accept size_t index parameter.

The remote users need the SUPER privilege because by default Spider sends a
'SET SQL_LOG_OFF' statement to the data nodes.  This is controlled by the
spider_internal_sql_log_off configuration setting on the Spider node, which
can only be set to 0 or 1, with a default value of 1.

I have fixed the problem by changing this configuration setting so that if it
is NOT SET, which is the most likely case, the Spider node DOES NOT SEND the
'SET SQL_LOG_OFF' statement to the data nodes.  However if the
spider_internal_sql_log_off setting IS EXPLICITLY SET to either 0 or 1, then
the Spider node DOES SEND the 'SET SQL_LOG_OFF' statement, requiring a remote
user with the SUPER privilege.  The Spider documentation will be updated to
reflect this change.

Author:
  Jacob Mathew.

Reviewer:
  Kentoku Shiba.

Cherry-Picked:
  Commit 72f0efac on branch bb-10.3-MDEV-15697

The remote users need the SUPER privilege because by default Spider sends a
'SET SQL_LOG_OFF' statement to the data nodes.  This is controlled by the
spider_internal_sql_log_off configuration setting on the Spider node, which
can only be set to 0 or 1, with a default value of 1.

I have fixed the problem by changing this configuration setting so that if it
is NOT SET, which is the most likely case, the Spider node DOES NOT SEND the
'SET SQL_LOG_OFF' statement to the data nodes.  However if the
spider_internal_sql_log_off setting IS EXPLICITLY SET to either 0 or 1, then
the Spider node DOES SEND the 'SET SQL_LOG_OFF' statement, requiring a remote
user with the SUPER privilege.  The Spider documentation will be updated to
reflect this change.

Author:
  Jacob Mathew.

Reviewer:
  Kentoku Shiba.

Merged:
  Branch bb-10.3-MDEV-15697 into 10.3

MDEV-16125 Crash or ASAN heap-buffer-overflow in mach_read_from_n_little_endian upon ALTER TABLE with blob

- Virtual column should be considered only to find the respective non-null fields.
But virtual column can never changed from NULL to NOT NULL.

Problem:
=======
InnoDB master thread encounters the shutdown state as SRV_SHUTDOWN_FLUSH_PHASE
when innodb_force_recovery >=2 and slow scheduling of master thread during
shutdown.

Fix:
====
There is no need for master thread itself if innodb_force_recovery >=2.
Don't create the master thread if innodb_force_recovery >= 2

The problem is hard to repeat, and I failed to create a deterministic
test case. Online index creation creates stubs for to-be-created indexes.
If index creation fails, we could remove these stubs while locks exist
in the indexes. (This would require that the index creation was completed,
and a concurrent DML operation acquired a lock on a record in the
uncommitted index. If a duplicate key error occurs in an uncommitted
index, the error will be reported for the CREATE UNIQUE INDEX, not for
the DML operation that tried to insert the duplicate.)

dict_table_try_drop_aborted(), row_merge_drop_indexes(): If transactional
locks exist on the table, keep the table->indexes intact.

ibuf_restore_pos(): Do not issue any messages if the tablespace
is being dropped or truncated. In MariaDB 10.2, TRUNCATE TABLE
would not change the tablespace ID, and therefore the tablespace
would be found, even though TRUNCATE is in progress.

Furthermore, do not commit suicide if restoring the change buffer
cursor fails. The worst that could happen is that a secondary index
becomes corrupted due to incomplete change buffer merge.

upon select with view and subqueries

This bug occurred when a splittable materialized derived/view
were used inside another splittable materialized derived/view.
The bug happened because the function JOIN::fix_all_splittings_in_plan()
was called at the very beginning of the optimization phase 2 at
the moment when the plan structure of the embedding derived/view
were not valid. The proper position for this call is the very
end of the optimization phase 1.

Remove the setup_ports function call. This is related to
https://github.com/MariaDB/server/pull/717

Thanks to Daniel Black and Bart S.

Debug command-line option should be loose, otherwise the test
fails before it has a chance to be skipped

Reasoning:
- Shorter and clearer code
- Better encapsulation
  (a fair number of Lex_input_stream methods and members were
   moved to the private section)

New methods:

  int lex_token(union YYSTYPE *yylval, THD *thd);
  bool consume_comment(int remaining_recursions_permitted);
  int lex_one_token(union YYSTYPE *yylval, THD *thd);
  int find_keyword(Lex_ident_cli_st *str, uint len, bool function);
  LEX_CSTRING get_token(uint skip, uint length);

Additional changes:

- Removing Lex_input_stream::yylval.
  In the original code it was just an alias
  for the "yylval" passed to lex_one_token().
  This coding style is bug prone and is hard to follow.
  In the new reduction "yylval" (or its components) is passed to
  the affected methods as a parameter.
- Moving the code in sql_lex.h up and down between "private" and "public"
  sections (sorry if this made the diff somewhat harder to read)

The reason is the missing HAVE_OPENSSL define for mariabackup.

The following variables are used in this project, but they are set to NOTFOUND.
LZ4_LIBS

The reason for the failure is that pkg_check_modules will not guarantee
<prefix>_LIBRARY_DIRS variable to be set, according to documentation.
When it's not set, we would force find_library to look in an empty path
and thus fail to correctly find LZ4_LIBS, although pck_check_modules
did previously discover that the library is installed.

To fix the problem and still keep the logic of first following
LIBLZ4_LIBRARY_DIRS and *then* look at other paths, we call find_library
twice. This is the recommended approach, according to CMake 3.11
documentation.

During rollback of temporary table logs, secondary index should delete mark
the index entry instead of removing it completely.