The code in the "sp_tail" rule in sql_yacc.yy always
used YYLIP->get_cpp_tok_start() as the start of the body,
and did not check for possible lookahead which happens
for keywords "FOR", "VALUES" and "WITH" for LALR(2)
resolution in Lex_input_stream::lex_token().

In case of the lookahead token presence,
get_tok_start_prev() should have been used instead
of get_cpp_tok_start() as the beginning of the SP body.

Change summary:

This patch hides the implementation of the lookahead
token completely inside Lex_input_stream.
The users of Lex_input_stream now just get token-by-token
transparently and should not care about lookahead any more.
Now external users of Lex_input_stream
are not aware of the lookahead token at all.

Change details:

- Moving Lex_input_stream::has_lookahead() into the "private" section.

- Removing Lex_input_stream::get_tok_start_prev() and
  Lex_input_stream::get_cpp_start_prev().

- Fixing the external code to call get_tok_start() and get_cpp_tok_start()
  in all places where get_tok_start_prev() and get_cpp_start_prev()
  where used.

- Adding a test for has_lookahead() right inside
  get_tok_start() and get_cpp_tok_start().
  If there is a lookahead token, these methods now
  return the position of the previous token automatically:

   const char *get_tok_start()
   {
     return has_lookahead() ? m_tok_start_prev : m_tok_start;
   }

   const char *get_cpp_tok_start()
   {
    return has_lookahead() ? m_cpp_tok_start_prev : m_cpp_tok_start;
   }

- Fixing the internal code inside Lex_input_stream methods
  to use m_tok_start and m_cpp_tok_start directly,
  instead of calling get_tok_start() and get_cpp_tok_start(),
  to make sure to access to the *current* token position
  (independently of a lookahead token presence).

The remote users need the SUPER privilege because by default Spider sends a
'SET SQL_LOG_OFF' statement to the data nodes.  This is controlled by the
spider_internal_sql_log_off configuration setting on the Spider node, which
can only be set to 0 or 1, with a default value of 1.

I have fixed the problem by changing this configuration setting so that if it
is NOT SET, which is the most likely case, the Spider node DOES NOT SEND the
'SET SQL_LOG_OFF' statement to the data nodes.  However if the
spider_internal_sql_log_off setting IS EXPLICITLY SET to either 0 or 1, then
the Spider node DOES SEND the 'SET SQL_LOG_OFF' statement, requiring a remote
user with the SUPER privilege.  The Spider documentation will be updated to
reflect this change.

Author:
  Jacob Mathew.

Reviewer:
  Kentoku Shiba.

Merged:
  Branch bb-10.3-MDEV-15697 into 10.3

MDEV-16125 Crash or ASAN heap-buffer-overflow in mach_read_from_n_little_endian upon ALTER TABLE with blob

- Virtual column should be considered only to find the respective non-null fields.
But virtual column can never changed from NULL to NOT NULL.

Problem:
=======
InnoDB master thread encounters the shutdown state as SRV_SHUTDOWN_FLUSH_PHASE
when innodb_force_recovery >=2 and slow scheduling of master thread during
shutdown.

Fix:
====
There is no need for master thread itself if innodb_force_recovery >=2.
Don't create the master thread if innodb_force_recovery >= 2

ibuf_restore_pos(): Do not issue any messages if the tablespace
is being dropped or truncated. In MariaDB 10.2, TRUNCATE TABLE
would not change the tablespace ID, and therefore the tablespace
would be found, even though TRUNCATE is in progress.

Furthermore, do not commit suicide if restoring the change buffer
cursor fails. The worst that could happen is that a secondary index
becomes corrupted due to incomplete change buffer merge.

upon select with view and subqueries

This bug occurred when a splittable materialized derived/view
were used inside another splittable materialized derived/view.
The bug happened because the function JOIN::fix_all_splittings_in_plan()
was called at the very beginning of the optimization phase 2 at
the moment when the plan structure of the embedding derived/view
were not valid. The proper position for this call is the very
end of the optimization phase 1.

Debug command-line option should be loose, otherwise the test
fails before it has a chance to be skipped

Reasoning:
- Shorter and clearer code
- Better encapsulation
  (a fair number of Lex_input_stream methods and members were
   moved to the private section)

New methods:

  int lex_token(union YYSTYPE *yylval, THD *thd);
  bool consume_comment(int remaining_recursions_permitted);
  int lex_one_token(union YYSTYPE *yylval, THD *thd);
  int find_keyword(Lex_ident_cli_st *str, uint len, bool function);
  LEX_CSTRING get_token(uint skip, uint length);

Additional changes:

- Removing Lex_input_stream::yylval.
  In the original code it was just an alias
  for the "yylval" passed to lex_one_token().
  This coding style is bug prone and is hard to follow.
  In the new reduction "yylval" (or its components) is passed to
  the affected methods as a parameter.
- Moving the code in sql_lex.h up and down between "private" and "public"
  sections (sorry if this made the diff somewhat harder to read)

During rollback of temporary table logs, secondary index should delete mark
the index entry instead of removing it completely.

The register keyword has no effect in C++, and it has been deprecated
in newer versions of the standard.

MDEV 13029 Assertion `ds_control' failed in debug_sync upon closing connection
after creating temporary sequence

This test doesn't fail anymore. Adding it to ensure that the bug doesn't
appear again.

Looks like the bug was fixed some time ago (at least I can't repeat it).
I added the test case just have this case tested properly.

MDEV-15843 mysys: remove optimized memcpy from 18 years ago

While this code has remained dormant for 18 years, libc implementers
have used assembly features to gain improvements using achitecture
features optimized and by the buffer length like:
* https://svnweb.freebsd.org/base/head/lib/libc/amd64/string/memcmp.S
* https://sourceware.org/git/?p=glibc.git;a=blob;f=sysdeps/aarch64/memcmp.S
* https://sourceware.org/git/?p=glibc.git;a=blob;f=sysdeps/powerpc/powerpc64/memcpy.S

From an sysbench-1.0.6 oltp_read_only test on binary charset table:

x86_64:
was ptr_compare_0:
perf report -g --no-children:
+    3.37%  mysqld   mysqld               [.] hp_rec_hashnr
+    3.15%  mysqld   libc-2.26.so         [.] __memmove_avx_unaligned_erms
+    2.73%  mysqld   mysqld               [.] row_search_mvcc
+    1.97%  mysqld   mysqld               [.] rec_get_offsets_func
+    1.24%  mysqld   mysqld               [.] ptr_compare_0
+    1.14%  mysqld   mysqld               [.] my_qsort2

After: __memcmp_avx2_movbe
+    3.42%  mysqld   mysqld               [.] hp_rec_hashnr
+    2.96%  mysqld   libc-2.26.so         [.] __memmove_avx_unaligned_erms
+    2.91%  mysqld   mysqld               [.] row_search_mvcc
+    2.13%  mysqld   mysqld               [.] rec_get_offsets_func
+    1.18%  mysqld   libc-2.26.so         [.] __memcmp_avx2_movbe
+    1.04%  mysqld   mysqld               [.] evaluate_join_record
+    1.02%  mysqld   mysqld               [.] my_qsort2

Power9:
Before: ptr_compare_0
+    4.24%  mysqld   mysqld               [.] _Z15row_search_mvccPh15page_cur_mode_tP14row_prebuilt_tmm
+    2.18%  mysqld   mysqld               [.] hp_rec_hashnr
+    2.07%  mysqld   mysqld               [.] _Z20rec_get_offsets_funcPKhPK12dict_index_tPmbmPP16mem_block_info_t
+    1.60%  mysqld   mysqld               [.] _ZL20evaluate_join_recordP4JOINP13st_join_tablei
+    1.20%  mysqld   mysqld               [.] _ZN11ha_innobase13general_fetchEPhjj
+    1.05%  mysqld   mysqld               [.] _ZN17Item_func_between15val_int_cmp_intEv
+    0.92%  mysqld   mysqld               [.] _Z40row_sel_field_store_in_mysql_format_funcPhPK17mysql_row_templ_tPKhm
+    0.91%  mysqld   mysqld               [.] _ZNK10Item_param6PValue7val_intEPK19Type_std_attributes
+    0.84%  mysqld   mysqld               [.] ptr_compare_0

After: __memcmp_power8
+    2.29%  mysqld           mysqld                  [.] _Z15row_search_mvccPh15page_cur_mode_tP14row_prebuilt_tmm
+    1.32%  mysqld           mysqld                  [.] hp_rec_hashnr
+    1.18%  swapper          [kernel.kallsyms]       [k] power_enter_stop
+    1.12%  mysqld           mysqld                  [.] _Z20rec_get_offsets_funcPKhPK12dict_index_tPmbmPP16mem_block_info_t
+    0.87%  mysqld           mysqld                  [.] _ZL20evaluate_join_recordP4JOINP13st_join_tablei
+    0.87%  mysqld           [kernel.kallsyms]       [k] ___bpf_prog_run
+    0.76%  mysqld           libc-2.26.so            [.] __memcmp_power8
+    0.68%  mysqld           mysqld                  [.] _ZN11ha_innobase13general_fetchEPhjj
+    0.58%  mysqld           mysqld                  [.] _ZN17Item_func_between15val_int_cmp_intEv

MDEV-14825 Assertion `col->ord_part' in row_build_index_entry_low upon ROLLBACK or DELETE with concurrent ALTER on partitioned table

If creating a secondary index fails (typically, ADD UNIQUE INDEX fails
due to duplicate key), it is possible that concurrently running UPDATE
or DELETE will access the index stub and hit the debug assertion.

It does not make any sense to keep updating an uncommitted index whose
creation has failed.

dict_index_t::is_corrupted(): Replaces dict_index_is_corrupted().
Also take online_status into account.

Replace some calls to dict_index_is_clust() with calls to
dict_index_t::is_primary().

(clang-3.8)
sql/sp.cc:2834:53: warning: implicit conversion of NULL constant to 'size_t' (aka 'unsigned long') [-Wnull-conversion]
    size_t prefix_length= dot ? dot - tmp.str + 1 : NULL;
           ~~~~~~~~~~~~~                            ^~~~
                                                    0

Remove the warning for InnoDB rebuilding table to add column FTS_DOC_ID.

Introduced new alter algorithm type called NOCOPY & INSTANT for
inplace alter operation.

NOCOPY - Algorithm refuses any alter operation that would
rebuild the clustered index. It is a subset of INPLACE algorithm.

INSTANT - Algorithm allow any alter operation that would
modify only meta data. It is a subset of NOCOPY algorithm.

Introduce new variable called alter_algorithm. The values are
DEFAULT(0), COPY(1), INPLACE(2), NOCOPY(3), INSTANT(4)

Message to deprecate old_alter_table variable and make it alias
for alter_algorithm variable.

alter_algorithm variable for slave is always set to default.

Compiles to same vector code, just a bit simplier.
vec_crc32.c is now identical to upstream
(https://github.com/antonblanchard/crc32-vpmsum/).

C code by Rogerio Alves <rogealve@br.ibm.com>

This implemention has been tested on big endian too.
Signed-off-by: Daniel Black <daniel@linux.ibm.com>

Signed-off-by: Daniel Black <daniel.black@au.ibm.com>

This is mostly to document that read_to_buffer can read more than 65K.
Also changed merge_buffers to return bool instead of int

As thd->alloc() and new automatically calls my_error(ER_OUTOFMEORY)
there is no reason to call mem_alloc_error()

Other things:
- Fixed bug in mysql_unpack_partition() where lex.part_info was
  changed even if it would be a null pointer

Performance schema likely/unlikely assume that performance schema
is enabled by default, which causes a performance degradation for
default installations that doesn't have performance schema enabled.

Fixed by changing the likely/unlikely in PS to assume it's
not enabled. This can be changed by compiling with
-DPSI_ON_BY_DEFAULT

Other changes:
- Added psi_likely/psi_unlikely that is depending on
  PSI_ON_BY_DEFAULT. psi_likely() is assumed to be true
  if PS is enabled.
- Added likely/unlikely to some PS interface code.
- Moved pfs_enabled to mysys (was initialized but not used before)
- Added "if (pfs_likely(pfs_enabled))" around calls to PS to avoid
  an extra call if PS is not enabled.
- Moved checking flag_global_instrumention before other flags
  to speed up the case when PS is not enabled.

To use:
- Compile with -DUSE_MY_LIKELY
- Change (with replace) all likely/unlikely to my_likely/my_/unlikely
  replace likely my_likely unlikely my_unlikely -- *c *h
- Start mysqld with -T
- run some test
- When mysqld has shut down cleanely, report will be on stderr

Added to:
- if (error)
- Lex
- sql_yacc.yy and sql_yacc_ora.yy
- In header files to alloc() calls
- Added thd argument to thd_net_is_killed()

It is test only (fix was done by Monty in ha_sequence::open by allocating ref)

            t1.pk IS NOT NULL where pk is a PRIMARY KEY

For equalites in the WHERE clause we create a keyuse array that contains the set of all equalities.
For each KEYUSE inside the keyuse array we have a field "null_rejecting"
which tells that the equality will not hold if either the left or right
hand side of the equality is NULL.
If the equality is NULL rejecting then we accordingly add a NOT NULL condition for the field present in
the item val(present in the KEYUSE struct) when we are doing ref access.

For the optimization of splitting with GROUP BY we always set the null_rejecting to TRUE and we are doing ref access on
the GROUP by field. This does create a problem when the equality is NOT NULL rejecting. This happens in this case as
in the equality we have the right hand side as t1.pk where pk is a PRIMARY KEY , hence it is NOT NULLABLE. So we should have
null rejecting set to FALSE for such a case.

The bug was the we copied the lock type to the underlying engine even when
external_lock failed.