These are helpful tools to quickly see what optimizer switch options
are on or off.  The different options are displayed alphabetically

MDEV-30310 Assertion failure in best_access_path upon IN exceeding IN_PREDICATE_CONVERSION_THRESHOLD, derived_with_keys=off

The bug was some old code that, without any explanation, reset
PART_KEY_FLAG from fields in temporary tables. This caused
join_tab->key_dependent to not be updated properly, which caused
an assert.

Added comments that not used keys of derivied tables will be deleted.
Added some comments about checking if pos_in_table_list is 0.

Other things:
- Added a marker (DBTYPE_IN_PREDICATE) in TABLE_LIST->derived_type
  to indicate that the table was generated from IN (list). This is
  useful for debugging and can later be used by explain if needed.
- Removed a not needed test of table->pos_in_table_list as it should
  always be valid at this point in time.

The problem was an assignment in test_quick_select() that flagged empty
tables with "Impossible where". This test was however wrong as it
didn't work correctly for left join.

Removed the test, but added checking of empty tables in DELETE and UPDATE
to get similar EXPLAIN as before.

The new tests is a bit more strict (better) than before as it catches all
cases of empty tables in single table DELETE/UPDATE.

Fixes also
MDEV-30104 Server crashes in handler_rowid_filter_check upon ANALYZE TABLE

cancel_pushed_rowid_filter() didn't inform the handler that rowid_filter
was canceled.

Fixed cost calculation for MERGE tables with 0 tables

The main difference in code path between EQ_REF and REF is that for
REF we have to do an extra read_next on the index to check that there
is no more matching rows.

Before this patch we added a preference of EQ_REF by ensuring that REF
would always estimate to find at least 2 rows.

This patch adds the cost of the extra key read_next to REF access and
removes the code that limited REF to at least 2 rows. For some queries
this can have a big effect as the total estimated rows will be halved
for each REF table with 1 rows.

multi_range cost calculations are also changed to take into account
the difference between EQ_REF and REF.

The effect of the patch to the test suite:
- About 80 test case changed
- Almost all changes where for EXPLAIN where estimated rows for REF
  where changed from 2 to 1.
- A few test cases using explain extended had a change of 'filtered'.
  This is because of the estimated rows are now closer to the
  calculated selectivity.
- A very few test had a change of table order.
  This is because the change of estimated rows from 2 to 1 or the small
  cost change for REF
  (main.subselect_sj_jcl6, main.group_by, main.dervied_cond_pushdown,
  main.distinct, main.join_nested, main.order_by, main.join_cache)
- No key statistics and the estimated rows are now smaller which cased
  estimated filtering to be lower.
  (main.subselect_sj_mat)
- The number of total rows are halved.
  (main.derived_cond_pushdown)
- Plans with 1 row changed to use RANGE instead of REF.
  (main.group_min_max)
- ALL changed to REF
  (main.key_diff)
- Key changed from ref + index_only to PRIMARY key for InnoDB, as
  OPTIMIZER_ROW_LOOKUP_COST + OPTIMIZER_ROW_NEXT_FIND_COST is smaller than
  OPTIMIZER_KEY_LOOKUP_COST + OPTIMIZER_KEY_NEXT_FIND_COST.
  (main.join_outer_innodb)
- Cost changes printouts
  (main.opt_trace*)
- Result order change
  (innodb_gis.rtree)

The old code counted selectivity double in case of queries like:
WHERE key_part1=1 and key_part2 < 100
if the optimizer would decide to use a REF access on key_part1.

The new code in best_access_path() that changes REF access to RANGE
if the RANGE key is longer makes this issue less likely to happen.

I was not able to create a test case for 11.0, however if one ports this
patch to a MariaDB version without the change of REF to RANGE, the
selectivity will be counted double.

The reason for this is that we call file->index_flags(index, 0, 1)
multiple times in best_access_patch()when optimizing a table.
For example, in InnoDB, the calls is not trivial (4 if's and 2 assignments)
Now the function is inlined and is just a memory reference.

Other things:
- handler::is_clustering_key() and pk_is_clustering_key() are now inline.
- Added TABLE::can_use_rowid_filter() to simplify some code.
- Test if we should use a rowid_filter only if can_use_rowid_filter() is
  true.
- Added TABLE::is_clustering_key() to avoid a memory reference.
- Simplify some code using the fact that HA_KEYREAD_ONLY is true implies
  that HA_CLUSTERED_INDEX is false.
- Added DBUG_ASSERT to TABLE::best_range_rowid_filter() to ensure we
  do not call it with a clustering key.
- Reorginized elements in struct st_key to get better memory alignment.
- Updated ha_innobase::index_flags() to not have
  HA_DO_RANGE_FILTER_PUSHDOWN for clustered index

- Increased timeout for binlog_mysqlbinlog_raw_flush.test.
  The old timeout was not enough when running with --valgrind
- Disabled ssl_timeout for --valgrind as it times out
- Disabled binlog_truncate_multi_engine for --valgrind as it does restarts

This could happen if mtr_grab_file() returned empty (happened to me)

Fixes over previous patches: do tracing of attached conditions
close to where we generate them.

Fix the tracing code to print the right conditions.

MDEV-21092,MDEV-21095,MDEV-29997: Optimizer Trace for index condition pushdown, partition pruning, exists-to-in

        Add Optimizer Tracing for:
        - Index Condition Pushdown
        - Partition Pruning
        - Exists-to-IN optimization

Make sure the queries use the intended query plan

This seems to confuse windows.

This was done after discussions with Igor, Sanja and Bar.

The main reason for removing the deprication was to ensure that MariaDB
is always backward compatible whenever possible.

Other things:
- Added statistics counters, mainly for the feedback plugin.
  - INTO OUTFILE
  - INTO variable
  - If INTO is using the old syntax (end of query)

In essence this means that we expect the user query to have at least
one matching row in the end.
This change will not affect the estimated rows for the plan, but will
ensure that the cost for adding a table is not neglected because of
record count being too low.

The reasons for this is that if we have table combination that
together has a very high selectivity then join record_count could
become very low (close to 0)

This would cause costs for all future tables to be so small that they
are irrelevant for the rest of the plan.
This has been shown to be the case in some performance benchmarks and
in a few mtr tests.

There is also still a problem in selectivity calculations as joining two
tables in different order causes a different estimation of total rows.
This can be seen in selectivity_innodb.test, test 'Q20' where joining
nation,supplier is expecting 1.111 rows_out while joining supplier,nation
is expecting 0.04 rows_out.

The reason for 0.04 is that the optimizer estimates 'supplier' to have
10 matching rows, and joining with nation (eq_ref) has 1 row. However
selectivity of n_name = 'UNITED STATES' makes the optimizer things
that there will be only 0.04 matching rows.

This patch avoids this "too low row count" to affect cost
caclulations.

- Changed 'WARNING' of type "You need to use --log-bin to make ... work"
  to 'Note'
- Only print startup Notes if log_warnings >= 4

"select * from information_schema.tables limit 1" was giving the following
warning in the log:

[ERROR] Invalid (old?) table or database name '#rocksdb'

Basic printout for join and table execution costs.

- Simplified test by setting read_time=DBL_MAX at start of loop if
  FORCE INDEX is used
- No need to test for 'group by' as the cost compare should handle it.
- Only one test change where index scan was replaced with table scan
 (correct)

- The comment in test_if_skip_sort_order was removed together with
  a not needed test of 'select'

Added override to a few functions in ha_partition.h

In the case one has an old Aria log file that ands with a Aria checkpoint
and the server restarts after next recovery, just after created a
new Aria log file (of 8K), the Aria recovery code would abort.
If one would try to delete all Aria log files after this (but not the
aria_control_file), the server would crash during recovery.

The problem was that translog_get_last_page_addr() would regard a log file
of exactly 8K as illegal and the rest of the code could not handle this
case.

Another issue was that if there was a crash directly after the log file
head was written to the next page, the code in translog_get_next_chunk()
would crash.

This patch fixes most of the issues, but not all. For Sanja to look at!

Things fixed:
- Added code to ignore 8K log files.
- Removed ASSERT in translog_get_next_chunk() that checks if page only
  contains the log page header.

I spent 4 hours on work and 12 hours of testing to try to find
the reason for aria crashing in recovery when starting a new test,
in which case the 'data directory' should be a copy of "install.db",
but aria_log.00000001 content was not correct.

The following changes are mostly done to make it a bit easier to find out
more in case of future similar crashes:

- Mark last_checkpoint_lsn volatile (safety).
- Write checkpoint message to aria_recovery.trace
- When compling with DBUG and with HAVE_DBUG_TRANSLOG_SRC,
  use checksum's for Aria log pages. We cannot have it on by default
  for DBUG servers yet as there is bugs when changing CRC between
  restarts.
- Added a message to mtr --verbose when copying the data directory.
- Removed extra linefeed in Aria recovery message (cleanup)

This includes:
- cleanup and optimization of filtering and pushdown engine code.
- Adjusted costs for rowid filters (based on extensive testing
  and profiling).

This made a small two changes to the handler_rowid_filter_is_active()
API:
- One should not call it with a zero pointer!
- One does not need to call handler_rowid_filter_is_active() for every
  row anymore. It is enough to check if filter is active by calling it
  call it during index_init() or when handler::rowid_filter_changed()
  is called

The changes was to avoid unnecessary function calls and checks if
pushdown conditions and rowid_filter is not used.

Updated costs for rowid_filter_lookup() to be closer to reality.
The old cost was based only on rowid_compare_cost. This is now
changed to take into account the overhead in checking the rowid.

Changed the Range_rowid_filter class to use DYNAMIC_ARRAY directly
instead of Dynamic_array<>. This was done to be able to use the new
append_dynamic() functions which gives a notable speed improvment
compared to the old code.  Removing the abstraction also makes
the code easier to understand.

The cost of filtering is now slightly lower than before, which
is reflected in some test cases that is now using rowid filters.

This helps with debugging as 'Query: ' in DBUG traces will show something
useful, for internal transactions, instead of just "".

This will speed up using tables that are already zerofilled
with aria_chk --zerofill.