GitLab

- are_tables_local() failed to recognize the fact that OUTER_REF_TABLE_BIT is ok 
  for SJ-Materialization. This caused zero-length ref access to be constructed, which
  led to an assert.

- Let join buffering code correctly take into account rowids needed 
  by DuplicateElimination when it is calculating minimum record sizes.
- In JOIN_CACHE::write_record_data, added asserts that prevent us from 
  writing beyond the end of the buffer.

For any query JOIN::optimize() should call the method
SELECT::save_leaf_tables after the last transformation
that utilizes the statement memory rather than the 
execution memory.
  

- Don't use join buffering for tables that are within ranges that are 
  covered by LooseScan strategy.

Fixed lp:828514 "Assertion `! is_set()' failed in Diagnostics_area::set_ok_status with derived table + subquery + concurrent DML"


sql/item_subselect.cc:
  Added check of error condtions (safety)
sql/sql_join_cache.cc:
  Added DBUG to some functions.
  Added error checking for calls to check_match(); This fixed the bug.
sql/sql_select.cc:
  Moved variable assignment to be close to where it's used (cleanup)

- Make JOIN::get_partial_cost_and_fanout() be able to handle join plans with 
  semi-join-materialization nests.

Analysis:
In the test query semi-join merges the inner-most subquery
into the outer subquery, and the optimization of the merged
subquery finds some new index access methods. Later the
IN-EXISTS transformation is applied to the unmerged subquery.
Since the optimizer is instructed to not consider
materialization, it reoptimizes the plan in-place to take into
account the new IN-EXISTS conditions. Just before reoptimization
JOIN::choose_subquery_plan resets the query plan, which also
resets the access methods found during the semi-join merge.
Then reoptimization discovers there are no new access methods,
but it leaves the query plan in its reset state. Later semi-join
crashes because it assumes these access methods are present.

Solution:
When reoptimizing in-place, reset the query plan only after new
access methods were discovered. If no new access methods were
discovered, leave the current plan as it was.

BUG##836491: Crash in Item_field::Item_field from add_ref_to_table_cond() with semijoin+materialization
- Let create_tmp_table set KEY_PART_INFO::fieldnr. It is needed in add_ref_to_table_cond(), and possibly other places.

Analysis:
Constant table optimization of the outer query finds that
the right side of the equality is a constant that can
be used for an eq_ref access to fetch one row from t1,
and substitute t1 with a constant. Thus constant optimization
triggers evaluation of the subquery during the optimize
phase of the outer query.

The innermost subquery requires a plan with a temporary
table because with InnoDB tables the exact count of rows
is not known, and the empty tables cannot be optimzied
way. JOIN::exec for the innermost subquery substitutes
the subquery tables with a temporary table.

When EXPLAIN gets to print the tables in the innermost
subquery, EXPLAIN needs to print the name of each table
through the corresponding TABLE_LIST object. However,
the temporary table created during execution doesn't
have a corresponding TABLE_LIST, so we get a null
pointer exception.

Solution:
The solution is to forbid using expensive constant
expressions for eq_ref access for contant table
optimization. Notice that eq_ref with a subquery
providing the value is still possible during regular
execution.

sql/sql_select.cc:
  items' cmp_type()'s must match, not result_type()'s

When the WHERE/HAVING condition of a subquery has been transformed
by the optimizer the pointer stored the 'where'/'having' field 
of the SELECT_LEX structure used for the subquery must be updated
accordingly. Otherwise the pointer may refer to an invalid item.
This can lead to the reported assertion failure for some queries
with correlated subqueries   

BUG#826935 Assertion `!table || (!table->read_set || bitmap_is_set(table->read_set, field_index))' failed
- add_ref_to_table_cond() should not just overwrite pre_idx_push_select_cond
  with the contents tab->select_cond.
  pre_idx_push_select_cond exists precisely for the reason that it may contain
  a condition that is a strict superset of what is in tab->select_cond. 
  The fix is to inject generated equality into pre_idx_push_select_cond.

- Make simplify_joins() set maybe_null=FALSE for tables that were on the 
  inner sides of inner joins and then were moved to the inner sides of semi-joins.

This bug is a special case of lp:813447.

Analysis:
Constant optimization finds that the condition t2.a = 1
can be used to access the primary key of table 't2'. As
a result both outer table t1,t2 are considered as constant
when we reach the execution phase. At the same time, during
constant optimization, the IN predicate is not evaluated
because it is expensive.

When execution of the outer query reaches do_select(),
control flow enter the branch:
if (join->table_count == join->const_tables)
{ ... }
This branch checks only the WHERE and HAVING clauses,
but doesn't check the ON clauses of the query. Since the
IN predicate was not evaluated during optimization, it is
not evaluated at all, thus execution doesn't detect that
the ON clause is FALSE.

Solution:
Similar to the patch for bug lp:813447, exclude system
tables from constant substitution based on unique key
lookups if there is an expensive ON condition on the
inner table.

- create_ref_for_key() has the code that walks KEYUSE array and tries to use
  maximum number of keyparts for ref (and eq_ref and ref_or_null) access.
  When one constructs ref access for table that is inside a SJ-Materialization
  nest, it is not possible to use tables that are ouside the nest (because 
  materialization is performed before they have any "current value").
  The bug was caused by this function not taking this into account.

This problem could be observed for queries with nested outer joins
for which the not_exist optimization were applicable. 
The problem was caused by the code of the patch for bug #49322
that erroneously forced the return to the previous nested loop
level when the join algorithm successfully builds a partial record
for an embedded outer to which the not_exist optimization could be
applied.
Actually the immediate return to the previous nested loops level
is correct only if this partial record is rejected by a predicate
pushed down to one of the inner tables of this outer join. Otherwise
attempts to find extensions of this record must be made.

the optimizer switch.

Do not make substitution of a single-row table if it is an inner
table of an outer join with on expression containing an expensive
subquery.

An aggregating query over an empty set of a join of two tables
with a rejecting HAVING clause erroneously could return a row.
It could happen in the cases when the optimizer made a conclusion
that the aggregating set was empty.
Wrong results were produced because the server missed initial
setting for aggregation functions in the mentioned cases.
      

BUG##806524: Assertion `join->best_read < 1.7976931348623157e+308 with table_elimination=on and derived_merge=on 
reset_nj_counters() used to rely on the fact that join nests have 
table->table==NULL. This ceased to be true wit new derived table
optimizations. Use test for table->nested_join!=NULL instead.

The problem was that optimizer removes some outer references (it they are
constant for example) and the list of outer items built during prepare phase is
not actual during execution phase when we need it as the cache parameters.
First solution was use pointer on pointer on outer reference Item and
initialize temporary table on demand. This solved most problem except case
when optimiser also reduce Item which contains outer references ('OR' in
this bug test suite).

The solution is to build the list of outer reference items on execution
phase (after optimization) on demand (just before temporary table creation)
by walking Item tree and finding outer references among Item_ident
(Item_field/Item_ref) and Item_sum items.

Removed depends_on list (because it is not neede any mnore for the cache, in the place where it was used it replaced with upper_refs).

Added processor (collect_outer_ref_processor) and get_cache_parameters() methods to collect outer references (or other expression parameters in future).

mysql-test/r/subselect_cache.result:
  A new test added.
mysql-test/r/subselect_scache.result:
  Changes in creating the cache and its paremeters order or adding arguments of aggregate function (which is a parameter also, but this has no influence on the result).
mysql-test/t/subselect_cache.test:
  Added a new test.
sql/item.cc:
  depends_on removed.
  
  Added processor (collect_outer_ref_processor) and get_cache_parameters() methods to collect outer references.
  
  Item_cache_wrapper collect parameters befor initialization of its cache.
sql/item.h:
  depends_on removed.
  
  Added processor (collect_outer_ref_processor) and get_cache_parameters() methods to collect outer references.
sql/item_cmpfunc.cc:
  depends_on removed.
  
  Added processor (collect_outer_ref_processor) to collect outer references.
sql/item_cmpfunc.h:
  Added processor (collect_outer_ref_processor) to collect outer references.
sql/item_subselect.cc:
  depends_on removed.
  Added processor get_cache_parameters() method to collect outer references.
sql/item_subselect.h:
  depends_on removed.
  Added processor get_cache_parameters() method to collect outer references.
sql/item_sum.cc:
  Added processor (collect_outer_ref_processor) method to collect outer references.
sql/item_sum.h:
  Added processor (collect_outer_ref_processor) and get_cache_parameters() methods to collect outer references.
sql/opt_range.cc:
  depends_on removed.
sql/sql_base.cc:
  depends_on removed.
sql/sql_class.h:
  New iterator added.
sql/sql_expression_cache.cc:
  Build of list of items resolved in outer query done just before creating expression cache on the first execution of the subquery which removes influence of optimizer removing items (all optimization already done).
sql/sql_expression_cache.h:
  Build of list of items resolved in outer query done just before creating expression cache on the first execution of the subquery which removes influence of optimizer removing items (all optimization already done).
sql/sql_lex.cc:
  depends_on removed.
sql/sql_lex.h:
  depends_on removed.
sql/sql_list.h:
  Added add_unique method to add only unique elements to the list.
sql/sql_select.cc:
  Support of new Item list added.
sql/sql_select.h:
  Support of new Item list added.

This bug could lead to wrong result sets for a query over a
materialized derived table or view accessed by a multi-component
key.
It happened because the function get_next_field_for_derived_key
was supposed to update its argument, and it did not do it.

Also:
1. simplified the code of the function mysql_derived_merge_for_insert.
2. moved merge of views/dt for multi-update/delete to the prepare stage.
3. the list of the references to the candidates for semi-join now is
   allocated in the statement memory.

Analysis:

For some of the re-executions of the correlated subquery the
where clause is false. In these cases the execution of the
subquery detects that it must generate a NULL row because of
implicit grouping. In this case the subquery execution reaches
the following code in do_select():

        while ((table= li++))
          mark_as_null_row(table->table);

This code marks all rows in the table as complete NULL rows.
In the example, when evaluating the field t2.f10 for the second
row, all bits of Field::null_ptr[0] are set by the previous call
to mark_as_null_row(). Then the call to Field::is_null()
returns true, resulting in a NULL for the MAX function.

Thus the lines above are not suitable for subquery re-execution
because mark_as_null_row() changes the NULL bits of each table
field, and there is no logic to restore these fields.

Solution:

The call to mark_as_null_row() was added by the fix for bug
lp:613029. Therefore removing the fix for lp:613029 corrects
this wrong result. At the same time the test for lp:613029
behaves correctly because the changes of MWL#89 result in a
different execution path where:
- the constant subquery is evaluated via JOIN::exec_const_cond
- detecting that it has an empty result triggers the branch
  if (zero_result_cause)
    return_zero_rows()
- return_zero_rows() calls mark_as_null_row().

The bitmap of used tables must be evaluated for the select list of every
materialized derived table / view and saved in a dedicated field.
This is also applied to materialized subqueries.

The value of THD::used tables should be re-evaluated after merges
of views and derived tables into the main query. 
Now it's done in the function SELECT_LEX::update_used_tables.
The re-evaluation of the 'used_table' bitmaps for the items
in HAVING, GROUP BY and ORDER BY clauses has been added as well.
 

The bug was caused by an incorrect code of the function
Item_direct_view_ref::replace_equal_field introduced in the
patch for bugs 717577, 724942. The function erroneously
returned the wrapped field instead of the Item_direct_view_ref
object itself in the cases when no replacement happened.

The bug masked two other minor bugs that could result in not
quite correct output of the EXPLAIN command for some queries.
They were fixed in the patch as well.
 

The function generate_derived_keys_for_table incorrectly handled
the cases when a materialized view or derived table could be accessed
by different keys on the same fields if these keys depended on the
same tables.
  

Analysis:
This bug consists of two related problems that are
result of too early evaluation of single-row subqueries
during the optimization phase of the outer query.

Several optimizer code paths try to evaluate single-row
subqueries in order to produce a constant and use that
constant for further optimzation.

When the execution of the subquery peforms destructive
changes to the representation of the subquery, and these
changes are not anticipated by the subsequent optimization
phases of the outer query, we tipically get a crash or
failed assert.

Specifically, in this bug the inner-most suqbuery with
DISTINCT triggers a substitution of the original JOIN
object by a single-table JOIN object with a temp table
needed to perform the DISTINCT operation (created by
JOIN::make_simple_join).

This substitution breaks EXPLAIN because:
a) in the first example JOIN::cleanup no longer can
reach the original table of the innermost subquery, and
close all indexes, and
b) in this second test query, EXPLAIN attempts to print
the name of the internal temp table, and crashes because
the temp table has no name (NULL pointer instead).

Solution:
a) fully disable subquery evaluation during optimization
in all cases - both for constant propagation and range
optimization, and
b) change JOIN::join_free() to perform cleanup irrespective
of EXPLAIN or not.

If no index is used to access a materialized derived table or view
then the value of TABLE_REF::key for this table must be (-1).

- JOIN::prepare would have set JOIN::table_count to incorrect value (bad merge of MWL 106)
- optimize_keyuse() would use table-bit as table number
  (the change in optimize_keyuse is also the reason for query plan changes. Not 
   expected to have much effect because only handles cases of no index statistics)
- st_select_lex::register_dependency_item() ignored the fact that some of the 
  selects on the dependency paths could have been merged to their parents (because they 
  were mergeable VIEWs)
- Undo the incorrect fix in Item_subselect::recalc_used_tables(): do not call 
  fix_after_pullout() for Item_subselect::Ref_to_outside members. 

If the expression for a derived table contained a clause LIMIT 0
SELECT from such derived table incorrectly returned a non-empty set.

Fixed by ensuring JOIN::do_send_rows to be updated after the call
of st_select_lex_unit::set_limit that sets the value of 
JOIN::unit->select_limit_cnt.

Due to this bug in the function generate_derived_keys_for_table some
key definitions to access materialized derived tables or materialized
views were constructed with invalid info for their key parts.
This could make the server crash when it optimized queries using 
materialized derived tables or materialized views. 

- The crash was because a NOT NULL table column inside the subquery was considered NULLable
  because the code thought it was on the inner side of an outer join nest.
- Fixed by making correct distinction between tables inside outer join nests and inside semi-join nests.

Split status variable Rows_read to Rows_read and Rows_tmp_read so that one can see how much real data is read.
Same was done with with Handler_update and Handler_write.
Fixed bug in MEMORY tables where some variables was counted twice.
Added new internal handler call 'ha_close()' to have one place to gather statistics.
Fixed bug where thd->open_options was set to wrong value when doing admin_recreate_table()


mysql-test/r/status.result:
  Updated test results and added new tests
mysql-test/r/status_user.result:
  Udated test results
mysql-test/t/status.test:
  Added new test for temporary table status variables
sql/ha_partition.cc:
  Changed to call ha_close() instead of close()
sql/handler.cc:
  Added internal_tmp_table variable for easy checking of temporary tables.
  Added new internal handler call 'ha_close()' to have one place to gather statistics.
  Gather statistics for internal temporary tables.
sql/handler.h:
  Added handler variables internal_tmp_table, rows_tmp_read.
  Split function update_index_statistics() to two.
  Added ha_update_tmp_row() for faster tmp table handling with more statistics.
sql/item_sum.cc:
  ha_write_row() -> ha_write_tmp_row()
sql/multi_range_read.cc:
  close() -> ha_close()
sql/mysqld.cc:
  New status variables: Rows_tmp_read, Handler_tmp_update and Handler_tmp_write
sql/opt_range.cc:
  close() -> ha_close()
sql/sql_base.cc:
  close() -> ha_close()
sql/sql_class.cc:
  Added handling of rows_tmp_read
sql/sql_class.h:
  Added new satistics variables.
  rows_read++  ->  update_rows_read() to be able to correctly count reads to internal temp tables.
  Added handler::ha_update_tmp_row()
sql/sql_connect.cc:
  Added comment
sql/sql_expression_cache.cc:
  ha_write_row() -> ha_write_tmp_row()
sql/sql_select.cc:
  close() -> ha_close()
  ha_update_row() -> ha_update_tmp_row()
sql/sql_show.cc:
  ha_write_row() -> ha_write_tmp_row()
sql/sql_table.cc:
  Fixed bug where thd->open_options was set to wrong value when doing admin_recreate_table()
sql/sql_union.cc:
  ha_write_row() -> ha_write_tmp_row()
sql/sql_update.cc:
  ha_write_row() -> ha_write_tmp_row()
sql/table.cc:
  close() -> ha_close()
storage/heap/ha_heap.cc:
  Removed double counting of statistic variables.
  close -> ha_close() to get tmp table statistics.
storage/maria/ha_maria.cc:
  close -> ha_close() to get tmp table statistics.

Ensured valid calculations of the estimates stored in JOIN_TAB::used_fieldlength.