Reasoning:
- Shorter and clearer code
- Better encapsulation
  (a fair number of Lex_input_stream methods and members were
   moved to the private section)

New methods:

  int lex_token(union YYSTYPE *yylval, THD *thd);
  bool consume_comment(int remaining_recursions_permitted);
  int lex_one_token(union YYSTYPE *yylval, THD *thd);
  int find_keyword(Lex_ident_cli_st *str, uint len, bool function);
  LEX_CSTRING get_token(uint skip, uint length);

Additional changes:

- Removing Lex_input_stream::yylval.
  In the original code it was just an alias
  for the "yylval" passed to lex_one_token().
  This coding style is bug prone and is hard to follow.
  In the new reduction "yylval" (or its components) is passed to
  the affected methods as a parameter.
- Moving the code in sql_lex.h up and down between "private" and "public"
  sections (sorry if this made the diff somewhat harder to read)

During rollback of temporary table logs, secondary index should delete mark
the index entry instead of removing it completely.

The register keyword has no effect in C++, and it has been deprecated
in newer versions of the standard.

MDEV 13029 Assertion `ds_control' failed in debug_sync upon closing connection
after creating temporary sequence

This test doesn't fail anymore. Adding it to ensure that the bug doesn't
appear again.

Looks like the bug was fixed some time ago (at least I can't repeat it).
I added the test case just have this case tested properly.

MDEV-15843 mysys: remove optimized memcpy from 18 years ago

While this code has remained dormant for 18 years, libc implementers
have used assembly features to gain improvements using achitecture
features optimized and by the buffer length like:
* https://svnweb.freebsd.org/base/head/lib/libc/amd64/string/memcmp.S
* https://sourceware.org/git/?p=glibc.git;a=blob;f=sysdeps/aarch64/memcmp.S
* https://sourceware.org/git/?p=glibc.git;a=blob;f=sysdeps/powerpc/powerpc64/memcpy.S

From an sysbench-1.0.6 oltp_read_only test on binary charset table:

x86_64:
was ptr_compare_0:
perf report -g --no-children:
+    3.37%  mysqld   mysqld               [.] hp_rec_hashnr
+    3.15%  mysqld   libc-2.26.so         [.] __memmove_avx_unaligned_erms
+    2.73%  mysqld   mysqld               [.] row_search_mvcc
+    1.97%  mysqld   mysqld               [.] rec_get_offsets_func
+    1.24%  mysqld   mysqld               [.] ptr_compare_0
+    1.14%  mysqld   mysqld               [.] my_qsort2

After: __memcmp_avx2_movbe
+    3.42%  mysqld   mysqld               [.] hp_rec_hashnr
+    2.96%  mysqld   libc-2.26.so         [.] __memmove_avx_unaligned_erms
+    2.91%  mysqld   mysqld               [.] row_search_mvcc
+    2.13%  mysqld   mysqld               [.] rec_get_offsets_func
+    1.18%  mysqld   libc-2.26.so         [.] __memcmp_avx2_movbe
+    1.04%  mysqld   mysqld               [.] evaluate_join_record
+    1.02%  mysqld   mysqld               [.] my_qsort2

Power9:
Before: ptr_compare_0
+    4.24%  mysqld   mysqld               [.] _Z15row_search_mvccPh15page_cur_mode_tP14row_prebuilt_tmm
+    2.18%  mysqld   mysqld               [.] hp_rec_hashnr
+    2.07%  mysqld   mysqld               [.] _Z20rec_get_offsets_funcPKhPK12dict_index_tPmbmPP16mem_block_info_t
+    1.60%  mysqld   mysqld               [.] _ZL20evaluate_join_recordP4JOINP13st_join_tablei
+    1.20%  mysqld   mysqld               [.] _ZN11ha_innobase13general_fetchEPhjj
+    1.05%  mysqld   mysqld               [.] _ZN17Item_func_between15val_int_cmp_intEv
+    0.92%  mysqld   mysqld               [.] _Z40row_sel_field_store_in_mysql_format_funcPhPK17mysql_row_templ_tPKhm
+    0.91%  mysqld   mysqld               [.] _ZNK10Item_param6PValue7val_intEPK19Type_std_attributes
+    0.84%  mysqld   mysqld               [.] ptr_compare_0

After: __memcmp_power8
+    2.29%  mysqld           mysqld                  [.] _Z15row_search_mvccPh15page_cur_mode_tP14row_prebuilt_tmm
+    1.32%  mysqld           mysqld                  [.] hp_rec_hashnr
+    1.18%  swapper          [kernel.kallsyms]       [k] power_enter_stop
+    1.12%  mysqld           mysqld                  [.] _Z20rec_get_offsets_funcPKhPK12dict_index_tPmbmPP16mem_block_info_t
+    0.87%  mysqld           mysqld                  [.] _ZL20evaluate_join_recordP4JOINP13st_join_tablei
+    0.87%  mysqld           [kernel.kallsyms]       [k] ___bpf_prog_run
+    0.76%  mysqld           libc-2.26.so            [.] __memcmp_power8
+    0.68%  mysqld           mysqld                  [.] _ZN11ha_innobase13general_fetchEPhjj
+    0.58%  mysqld           mysqld                  [.] _ZN17Item_func_between15val_int_cmp_intEv

MDEV-14825 Assertion `col->ord_part' in row_build_index_entry_low upon ROLLBACK or DELETE with concurrent ALTER on partitioned table

If creating a secondary index fails (typically, ADD UNIQUE INDEX fails
due to duplicate key), it is possible that concurrently running UPDATE
or DELETE will access the index stub and hit the debug assertion.

It does not make any sense to keep updating an uncommitted index whose
creation has failed.

dict_index_t::is_corrupted(): Replaces dict_index_is_corrupted().
Also take online_status into account.

Replace some calls to dict_index_is_clust() with calls to
dict_index_t::is_primary().

(clang-3.8)
sql/sp.cc:2834:53: warning: implicit conversion of NULL constant to 'size_t' (aka 'unsigned long') [-Wnull-conversion]
    size_t prefix_length= dot ? dot - tmp.str + 1 : NULL;
           ~~~~~~~~~~~~~                            ^~~~
                                                    0

Remove the warning for InnoDB rebuilding table to add column FTS_DOC_ID.

Introduced new alter algorithm type called NOCOPY & INSTANT for
inplace alter operation.

NOCOPY - Algorithm refuses any alter operation that would
rebuild the clustered index. It is a subset of INPLACE algorithm.

INSTANT - Algorithm allow any alter operation that would
modify only meta data. It is a subset of NOCOPY algorithm.

Introduce new variable called alter_algorithm. The values are
DEFAULT(0), COPY(1), INPLACE(2), NOCOPY(3), INSTANT(4)

Message to deprecate old_alter_table variable and make it alias
for alter_algorithm variable.

alter_algorithm variable for slave is always set to default.

This is mostly to document that read_to_buffer can read more than 65K.
Also changed merge_buffers to return bool instead of int

As thd->alloc() and new automatically calls my_error(ER_OUTOFMEORY)
there is no reason to call mem_alloc_error()

Other things:
- Fixed bug in mysql_unpack_partition() where lex.part_info was
  changed even if it would be a null pointer

Performance schema likely/unlikely assume that performance schema
is enabled by default, which causes a performance degradation for
default installations that doesn't have performance schema enabled.

Fixed by changing the likely/unlikely in PS to assume it's
not enabled. This can be changed by compiling with
-DPSI_ON_BY_DEFAULT

Other changes:
- Added psi_likely/psi_unlikely that is depending on
  PSI_ON_BY_DEFAULT. psi_likely() is assumed to be true
  if PS is enabled.
- Added likely/unlikely to some PS interface code.
- Moved pfs_enabled to mysys (was initialized but not used before)
- Added "if (pfs_likely(pfs_enabled))" around calls to PS to avoid
  an extra call if PS is not enabled.
- Moved checking flag_global_instrumention before other flags
  to speed up the case when PS is not enabled.

To use:
- Compile with -DUSE_MY_LIKELY
- Change (with replace) all likely/unlikely to my_likely/my_/unlikely
  replace likely my_likely unlikely my_unlikely -- *c *h
- Start mysqld with -T
- run some test
- When mysqld has shut down cleanely, report will be on stderr

Added to:
- if (error)
- Lex
- sql_yacc.yy and sql_yacc_ora.yy
- In header files to alloc() calls
- Added thd argument to thd_net_is_killed()

It is test only (fix was done by Monty in ha_sequence::open by allocating ref)

            t1.pk IS NOT NULL where pk is a PRIMARY KEY

For equalites in the WHERE clause we create a keyuse array that contains the set of all equalities.
For each KEYUSE inside the keyuse array we have a field "null_rejecting"
which tells that the equality will not hold if either the left or right
hand side of the equality is NULL.
If the equality is NULL rejecting then we accordingly add a NOT NULL condition for the field present in
the item val(present in the KEYUSE struct) when we are doing ref access.

For the optimization of splitting with GROUP BY we always set the null_rejecting to TRUE and we are doing ref access on
the GROUP by field. This does create a problem when the equality is NOT NULL rejecting. This happens in this case as
in the equality we have the right hand side as t1.pk where pk is a PRIMARY KEY , hence it is NOT NULLABLE. So we should have
null rejecting set to FALSE for such a case.

The bug was the we copied the lock type to the underlying engine even when
external_lock failed.

log_sys_init(), log_buffer_extend(): Add TRASH_ALLOC() instrumentation

log_write_up_to(): Correctly calculate the byte offset.

The number of records in INFORMATION_SCHEMA.COLUMNS depends on the
build options, and could easily change when features are added.
We are not interested in the number of rows returned. The test was
originally added because of problem 15 reported in MDEV-13900
(testing for MDEV-11369 instant ADD COLUMN). The issue was an
assertion failure ut_ad(!rec_is_default_row(rec, index))
in lock_clust_rec_cons_read_sees(), because the 'default row' record
was not being properly ignored by the b-tree cursor.

The fix for this bug was automatically merged from 10.2.  However, that fix
was incomplete in 10.3.  This commit is for the additional changes that are
necessary in 10.3.

Author:
  Jacob Mathew.

Reviewer:
  Kentoku Shiba.

Fixed by removing the check of single lock in sequence insert and let MDL
code handle deadlock detection

MDEV-15060 Assertion in row_log_table_apply_op after instant ADD when the table is emptied during subsequent ALTER TABLE

During an online table rebuild, a table could be emptied and converted
from 'instant ADD' format to plain (pre-10.3) format. All online_log
records for rebuilding the table must be written and parsed in the
format of the table that existed at the start of the operation.

row_log_t::n_core_fields: A new field for recording index->n_core_fields
when online ALTER is initiated in row_log_allocate().

row_log_t::is_instant(): Determine if the log is in the instant format.
Only invoked by the row_log_table_ family of functions.

dict_index_t::get_n_nullable(): Remove is_instant() debug assertions.
Because a table can be converted to non-instant format during a
table-rebuilding ALTER TABLE, these assertions would be bogus when
executing row_log_table_apply().

rec_init_offsets_temp(): Add the parameter n_core for passing the
original index->n_core_fields.

rec_init_offsets_temp(): Add a 3-parameter variant.

rec_init_offsets_comp_ordinary(): Add the parameter n_core for
passing the index->n_core_fields.

MDEV-12266 changed dict_table_t::space to a pointer.
Displaying pointer values in error messages would be even more
meaningless than displaying numeric tablespace identifiers.

row_create_table_for_mysql(): Do not display table->space when
deleting the file fails. We cannot dereference table->space here,
because fil_delete_tablespace() would have freed the object.

fil_wait_crypt_bg_threads(): Do not display table->space. We could
display table->space_id here, but it should not really add any value,
because the table reference-counts have no direct connection to files
or tablespaces.

btr_pcur_store_position(): Assert that the 'default row' record never
is the only record in a page. (If that would happen, an empty
root page would be re-created in the non-instant format, not containing
the special record.) When the cursor is positioned on the page infimum,
never use the 'default row' as the BTR_PCUR_BEFORE reference.
(This is additional cleanup, not fixing the bug.)

rec_copy_prefix_to_buf(): When converting a record prefix to
the non-instant-add format, copy the original number of null flags.
Rename the variable instant_len to instant_omit, and introduce a
few more variables to make the code easiser to read.

Note: In purge, rec_copy_prefix_to_buf() is also used for storing the
persistent cursor position on a 'default row' record. The stored record
reference will be garbage, but row_search_on_row_ref() will do special
handling to reposition the cursor on the 'default row', based on
ref->info_bits.

innodb.dml_purge: Also cover the 'default row'.

MDEV-16071 Server crashed in innobase_build_col_map / prepare_inplace_alter_table_dict or Assertion `tuple' failed in dtuple_get_nth_field upon altering table with virtual column

- Virtual column should be considered during innobase_build_col_map() to find out
whether the field changed from NULL to NOT NULL.

- Added two new test case for it.

When a comma separator is missing between COMMENT fields, Spider ignores the
parameter values that are beyond the last expected parameter value.  There are
also some error messages that Spider does generate on COMMENT fields that are
incorrectly formed.

I have introduced additional infrastructure in Spider to fix these problems.

Author:
  Jacob Mathew.

Reviewer:
  Kentoku Shiba.

Cherry-Picked:
  Commit c10da98b on branch bb-10.3-MDEV-15698

When a comma separator is missing between COMMENT fields, Spider ignores the
parameter values that are beyond the last expected parameter value.  There are
also some error messages that Spider does generate on COMMENT fields that are
incorrectly formed.

I have introduced additional infrastructure in Spider to fix these problems.

Author:
  Jacob Mathew.

Reviewer:
  Kentoku Shiba.

Merged From:
  Branch bb-10.3-MDEV-15698

Remove unused InnoDB function parameters and functions.

i_s_sys_virtual_fill_table(): Do not allocate heap memory.

mtr_is_block_fix(): Replace with mtr_memo_contains().

mtr_is_page_fix(): Replace with mtr_memo_contains_page().

mysql_test_db.sql is in the srcdir

MDEV-15712: If remote server used by Spider table is unavailable, some operations hang for a long time

When an attempt to connect to the remote server fails, Spider retries to
connect to the remote server 1000 times or until the connection attempt
succeeds.  This is perceived as a hang if the remote server remains
unavailable.

I have introduced changes in Spider's table status handler to fix this problem.

Author:
  Jacob Mathew.

Reviewer:
  Kentoku Shiba.

Cherry-Picked:
  Commit 6ee6933a on branch bb-10.3-MDEV-15712