- fixed incorrect query plans that resulted from emptying the
  keyuse array with the access methods of the non-modified query
  plan.
- fixed compiler warning.

sql/item.cc:
  Wrapper added.
sql/item.h:
  Wrapper added.
sql/mysql_priv.h:
  Wrap function added.
sql/sql_base.cc:
  Wrap function added.
  Fix of problem with WHERE consist of alone outer reference field by wrapping it.
sql/sql_select.cc:
  Fix of problem with HAVING consist of alone outer reference field by wrapping it.

Analysis:
Close to its end JOIN::optimize() assigns having to tmp_having, and
sets the having clause to NULL:

  tmp_having= having;
  if (select_options & SELECT_DESCRIBE)
  {
    error= 0;
    DBUG_RETURN(0);
  }
  having= 0;

At the same time, this query detects an empty result set, and calls
return_zero_rows(), which checks the HAVING clause as follows:

    if (having && having->val_int() == 0)
      send_row=0;

However having has been already set to NULL, so return_zero_rows
doesn't check the having clause, hence the wrong result.

Solution:
Check join->tmp_having in addition to join->having.

There is no additional test case, because the failure was in
the current regression test.

This fixes the wrong result in tests like compress, join, join_cache, greedy_optimizer and select_pkeycache

  
- Post-review fixes. Intermediate commit to address review point 1.6.
- Fixed valgrind warnings

Post-review fixes. Intermediate commit to address
review points 1.1, 1.2, 1.3, 1.4, 1.5, and 3.1, 3.2, 3.3.

Analysis:
The assert failed because st_select_lex::print() was called for subqueries
as follows:

Item_subselect::print() ->
  subselect_single_select_engine::print() -> st_select_lex::print()

It was Item_subselect::fix_fields() that set the thd by calling set_thd(),
so when this print() was called before fix_fields(), subselect_engine::thd
was NULL.

Solution:
The patch makes all constructors of all subselect_engine classes to take
a THD parameter. The default subselect_single_select_engine engine is created
early during parse time, in the Item_subselect::init call, so we pass the
correct THD object already at this point.

  
Analysis:
The fix for LP BUG#680846 avoids evaluation of constant expressions
with subqueries in the GROUP/ORDER clauses in the procedure
remove_const(). The purpose of remove_const is to remove constant
expressions in the GROUP/ORDER clauses.
  
In order delay until execution the evaluation of such subqueries,
they were not removed in the GROUP/ORDER clause. As a result temp
table creation during execution attempted to create a column in the
temp table for each constant GROUP/ORDER expression. However, the
logic in create_tmp_table is to not create temp table columns for
constant items. The crash was due to a group Item without a
corresponding column in the temp table for GROUP BY.
  
Solution:
The patch adds back removal of constant expressions with subqueries.
In order for such expressions to be evaluated, so that the server can
ensure that such subquries return 1 row, the evaluation of these
expressions is delayed until execution.

The second (final) patch.

Changed rows_read and rows_sent status variables to be longlong (to avoid compiler warnings)


sql/item_func.cc:
  Fixed wrong usage of alias
sql/item_subselect.cc:
  Changed buffer size to ulonglong to be able detect buffers bigger than size_t
sql/item_subselect.h:
  Changed buffer size to ulonglong to be able detect buffers bigger than size_t
sql/multi_range_read.cc:
  Fixed compiler warning by using correct type for function argument
sql/mysqld.cc:
  Changed rows_read and rows_sent status variables to be longlong
sql/opt_subselect.h:
  Fixed compiler warning by using correct type for function argument
sql/sql_class.cc:
  Changed rows_read and rows_sent status variables to be longlong
sql/sql_class.h:
  Changed rows_read and rows_sent status variables to be longlong
  Changed max_nulls_in_row to uint as this is number of columns in row.
  This fixed some compiler warnings.
sql/sql_select.cc:
  Added casts to avoid compiler warnings
storage/heap/ha_heap.cc:
  Initilize different types separate
storage/oqgraph/ha_oqgraph.cc:
  Fixed argument to store(longlong) to avoid compiler warnings

- Changed Cached_item_field not copy data for fields with NULL value
- In key_copy() and key_restore() don't copy data for fields with NULL value

Fixed code to avoid valgrind warnings
- Use c_ptr_safe instead of c_ptr()

Removed "QQ" from comments (QQ was ment to be used for internal comments that should be removed before pushing)
Fixed wrong alias used (from previous patch)


sql/event_db_repository.cc:
  Update testing if event table is valid (to avoid valgrind errors)
sql/ha_partition.cc:
  m_ordered_scan_ongoing was not initialized
  Reset null bits in record to avoid valgrind errors
sql/handler.h:
  Added flag if storage engine will write row verbatim and the row contains varchar or null fields
  (in which case we must clear the row to avoid valgrind warnings)
sql/item_buff.cc:
  Changed Cached_item_field not copy data for fields with NULL value
  (Optimization and avoids valgrind warnings)
sql/item_func.cc:
  c_ptr() -> c_ptr_safe()
sql/key.cc:
  In key_copy() and key_restore() don't copy data for fields with NULL value
sql/opt_range.cc:
  c_ptr() -> c_ptr_safe()
sql/sql_base.cc:
  Added TRASH() to table->record[0] to find out if we access not initialzed data.
  Initialize null_bytes to:
  - Get consistent tests
  - Ensure we don't get valgrind warnings for null fields (as we may only update a couple of bits in a byte)
sql/sql_class.cc:
  Removed "QQ" from comments
sql/sql_insert.cc:
  Initialize row to default values if we are using valgrind and row will be copied verbatim to disk in storage engine.
sql/sql_load.cc:
  QQ -> TODO
sql/sql_parse.cc:
  Removed old not used code marked QQ and withing "#ifdef REMOVED"
sql/sql_select.cc:
  QQ -> TODO
  Initialize some variables that was used uninitialized
  Added DBUG_ASSERT() to find out if thd was not properly initialized for sub queries
sql/sql_test.cc:
  Fixed format for printing to DBUG file
  Fixed wrong alias used (from previous patch)
sql/sql_trigger.h:
  QQ -> TODO
sql/table.cc:
  QQ -> TODO
storage/maria/ha_maria.cc:
  Mark table with HA_RECORD_MUST_BE_CLEAN_ON_WRITE, if row is written verbatim to disk and contains varchar or null fields.
storage/maria/ma_open.c:
  Added flags if table has varchar or null fields
storage/maria/maria_def.h:
  Added flags if table has varchar or null fields
storage/myisam/ha_myisam.cc:
  Mark table with HA_RECORD_MUST_BE_CLEAN_ON_WRITE, if row is written verbatim to disk and contains varchar or null fields.
storage/myisam/mi_open.c:
  Fixed memory overrun bug when using fulltext keys
storage/xtradb/row/row0sel.c:
  Removed initialization of null bits. (not needed anymore)

Analysis:
JOIN::optimize performs constant optimization of GROUP by clauses
by calling remove_const():
    group_list= remove_const(this, (old_group_list= group_list), conds,
                             rollup.state == ROLLUP::STATE_NONE,
			     &simple_group);
If it turns out that a GROUP clause references a field that is
computed by a single-row subquery, then the said optimization
performs premature execution of the subquery referenced by the
group clause.

Solution:
Block the evaluation of subqueries similarly to the approach
for the WHERE and JOIN..ON clauses.

Introduced a new flag in the class Item. The flag is set
to 1 only for items that are used in GROUP BY lists of
queries with ROLLUP.

Fix MySQL BUG#52344 - Subquery materialization: Assertion if subquery in on-clause of outer join

Original fix and comments from Oysten, adjusted for the different
subquery optimization in MariaDB.
"
Problem: If tables of an outer join are constant tables,
the associated on-clause will be evaluated in the optimization
phase. If the on-clause contains a query that is to be
executed with subquery materialization, this will not work
since the infrastructure for such execution is not yet set up.
      
Solution: Do not evaluate on-clause in optimization phase if
is_expensive() returns true for this clause.  This is how the
problem is currently avoided for where-clauses.  This works
because, Item_in_subselect::is_expensive_processor returns true
if query is to be executed with subquery materialization.
"
In addition, after MWL#89, in MariaDB if the IN-EXISTS strategy
is chosen, the in-to-exists predicates are insterted after
join_read_const_table() is called, resulting in evaluation of
the subquery without the in-to-exists predicates.

- Changed TABLE->alias to String to get fewer reallocs when alias are used.
- Preallocate some buffers

Changed some String->c_ptr() -> String->ptr() when \0 is not needed.
Fixed wrong usage of String->ptr() when we need a \0 terminated string.
Use my_strtod() instead of my_atof() to avoid having to add \0 to string.
c_ptr() -> c_ptr_safe() to avoid warnings from valgrind.
zr 

sql/event_db_repository.cc:
  Update usage of TABLE->alias
sql/event_scheduler.cc:
  c_ptr() -> c_ptr_safe()
sql/events.cc:
  c_ptr() -> ptr() as \0 was not needed
sql/field.cc:
  Update usage of TABLE->alias
sql/field.h:
  Update usage of TABLE->alias
sql/ha_partition.cc:
  Update usage of TABLE->alias
sql/handler.cc:
  Update usage of TABLE->alias
  Fixed wrong usage of str.ptr()
sql/item.cc:
  Fixed error where code wrongly assumed string was \0 terminated.
sql/item_func.cc:
  c_ptr() -> c_ptr_safe()
  Update usage of TABLE->alias
sql/item_sum.h:
  Use my_strtod() instead of my_atof() to avoid having to add \0 to string
sql/lock.cc:
  Update usage of TABLE->alias
sql/log.cc:
  c_ptr() -> ptr() as \0 was not needed
sql/log_event.cc:
  c_ptr_quick() -> ptr() as \0 was not needed
sql/opt_range.cc:
  ptr() -> c_ptr() as \0 is needed
sql/opt_subselect.cc:
  Update usage of TABLE->alias
sql/opt_table_elimination.cc:
  Update usage of TABLE->alias
sql/set_var.cc:
  ptr() -> c_ptr() as \0 is needed
  c_ptr() -> c_ptr_safe()
sql/sp.cc:
  c_ptr() -> ptr() as \0 was not needed
sql/sp_rcontext.cc:
  
  Update usage of TABLE->alias
sql/sql_base.cc:
  Preallocate buffers
  Update usage of TABLE->alias
sql/sql_class.cc:
  Fix arguments to sprintf() to work even if string is not \0 terminated
sql/sql_insert.cc:
  Update usage of TABLE->alias
  c_ptr() -> ptr() as \0 was not needed
sql/sql_load.cc:
  Preallocate buffers
  Trivial optimizations
sql/sql_parse.cc:
  Trivial optimization
sql/sql_plugin.cc:
  c_ptr() -> ptr() as \0 was not needed
sql/sql_select.cc:
  Update usage of TABLE->alias
sql/sql_show.cc:
  Update usage of TABLE->alias
sql/sql_string.h:
  Added move() function to move allocated memory from one object to another.
sql/sql_table.cc:
  Update usage of TABLE->alias
  c_ptr() -> c_ptr_safe()
sql/sql_test.cc:
  ptr() -> c_ptr_safe()
sql/sql_trigger.cc:
  Update usage of TABLE->alias
  c_ptr() -> c_ptr_safe()
sql/sql_update.cc:
  Update usage of TABLE->alias
sql/sql_view.cc:
  ptr() -> c_ptr_safe()
sql/sql_yacc.yy:
  ptr() -> c_ptr()
sql/table.cc:
  
  Update usage of TABLE->alias
sql/table.h:
  Changed TABLE->alias to String to get fewer reallocs when alias are used.
storage/federatedx/ha_federatedx.cc:
  Use c_ptr_safe() to ensure strings are \0 terminated.
storage/maria/ha_maria.cc:
  Update usage of TABLE->alias
storage/myisam/ha_myisam.cc:
  Update usage of TABLE->alias
storage/xtradb/row/row0sel.c:
  Ensure that null bits in record are properly reset.
  (Old code didn't work as row_search_for_mysql() can be called twice while reading fields from one row.

Analysis:
This another instance of the problem fixed in LP BUG#675981 -
evaluation of subqueries during EXPLAIN when the subquery plan
is incomplete because JOIN::optimize() generally doesn't create
complete execution plans for EXPLAIN statements.

In this case the call path is:
mysql_explain_union -> outer_join.exec -> outer_join.init_execution ->
create_sort_index -> filesort -> find_all_keys ->
SQL_SELECT::skip_record -> outer_where_clause.val_int -> ...
-> subselect_join.exec -> ... -> sub_select_cache

When calling sub_select_cache JOIN_TAB::cache is NULL because the cache
objects are not created for EXPLAIN statements.

Solution:
Delay the call to init_execution() after all EXPLAIN related processing
is completed. Thus init_execution() is not called at all during EXPLAIN.

Cause:
The optimize() phase for the subquery selected to use join buffering via setting
JOIN_TAB::next_select= sub_select_cache in make_join_readinfo, however, the call
to check_join_cache_usage() from make_join_readinfo didn't create the corresponding
JOIN_CACHE_BNL object because of the condition:
    if ((options & SELECT_DESCRIBE) ||
        (((tab->cache= new JOIN_CACHE_BNL(join, tab, prev_cache))) &&
         !tab->cache->init()))
Since EXPLAIN for subqueries runs regular execution, the constant predicates that
were delayed to be evaluated at the exec() phase, were evaluated during EXPLAIN.
As a result the outer JOIN::exec called JOIN::exec for the subquery, while the
subquery execution plan was no properly created, which resulted in an failed ASSERT.

Fix:
The patch blocks evaluation of constant expensive conditions during EXPLAIN. Notice
that these conditions are "constant" with respect to the outer query, thus in
general they could be arbitrarily expensive, which may result in very slow EXPLAINs.

The condition that was supposed to check whether a join table
is an inner table of a nested outer join or semi-join was not
quite correct in the code of the function check_join_cache_usage.
That's why some queries with nested outer joins triggered 
an assertion failure.
Encapsulated this condition in the new method called
JOIN_TAB::is_nested_inner and provided a proper code for it.

Also corrected a bug in the code of check_join_cache_usage()
that caused a downgrade of not first join buffers of the
level 5 and 7 to level 4 and 6 correspondingly.

The cause for the bug was two-fold:
1. Incorrect detection of whether a table is the first one in a query plan -
  "used_table & 1" actually checks if used_table is table with number "1".
2. Missing logic to delay the evaluation of (expensive) constant conditions
  during the execution phase.

The fix adds/changes:
The patch:
- removes incorrect treatment of expensive predicates from make_cond_for_table,
  and lets the caller decide when to evaluate expensive predicates.
- saves expensive constant conditions in JOIN::exec_const_cond,
  which is evaluated once in the beginning of JOIN::exec.

When pushing the condition for a table in the function
JOIN_TAB::make_scan_filter the optimizer must not push
conditions from WHERE if the table is some inner table
of an outer join..

The condition over outer tables extracted from the on expression
for a outer join must be ANDed to the condition pushed to the
first inner table of this outer join only.
Nested outer joins cannot use flat join buffers. So if join_cache_level
is set to 1 then any join algorithm employing join buffers cannot be used
for nested outer joins.

A non-incremental join buffer cannot be used for inner tables of nested
outer joins. That's why when join_cache_level is set to 7 it must
be downgraded to level 6 for the inner tables of nested outer joins.
For the same reason with join_cache_level set to 3 no join buffer is
used for the inner tables of outer joins (we could downgrade it to
level 2, but this level does not support ref access).

Miscalculation of the minimum possible buffer size could trigger
an assert in JOIN_CACHE_HASHED::put_record when if join_buffer_size
was set to the values that is less than the length of one record to
stored in the join buffer.
It happened due to the following mistakes:
- underestimation of space needed for a key in the hash table
  (we have to take into account that hash table can have more
  buckets than the expected number of records).
- the value of maximum total length of all records stored in
  the join buffer was not saved in the field max_used_fieldlength
  by the function calc_used_field_length.

The pushdown condition for the sorted table in a query can be complemented
by the conditions from HAVING. This transformation is done in JOIN::exec
pretty late after the original pushdown condition have been saved in the
field pre_idx_push_select_cond for the sorted table. So this field must
be updated after the inclusion of the condition from HAVING.

Avoid doing reallocs
Prealloc some strings / provide extension allocation size to some strings
This gave a 25 % speedup in some mysql-test-run tests.



mysys/safemalloc.c:
  More DBUG_PRINT
sql/net_serv.cc:
  Make all mallocs() look the similar. (just-for-safety fix)
sql/protocol.cc:
  Ensure that communication packet buffer is allocated.
  (It's freed by stored precedures and some DLL statements)
sql/sp.cc:
  Fixed valgrind warning
sql/sql_select.cc:
  Set extent allocation for buffer that has a lot of append() calls.
sql/sql_show.cc:
  Fixed wrong usage of string buffer. Old code worked in test suite 'just-by-chance'
sql/sql_string.cc:
  Call realloc_with_extra_if_needed() in append() functions.
sql/sql_string.h:
  Added 'extra_alloc' member, to specify chunck size for realloc().
  extra_alloc is addaptive to catch cases where preallocation of buffers is not done properly.
  Simplified free() to allow compiler to optimize things better (and to keep things consistent).
  Fixed shrink() to take into account the extra memory added to the Alloced_length in realloc(). This saves us a realloc() per query.
sql/sql_test.cc:
  Set extent allocation for buffer that has a lot of append() calls.
sql/table.cc:
  Set extent allocation for buffer that has a lot of append() calls.

try to find a match in the join buffer. It makes sense to check for a match
only those records satisfying WHERE/ON conditions that can be pushed to
the scanned table. It allows us to discard early some join candidates.

Such pushdown conditions were built when BNL join algorithm was employed,
but for they were not built when BNLH algorithm was used.
The patch removes this shortcoming. 

plans or wrong results due to the fact that JOIN_CACHE functions
ignored the possibility of interleaving materialized semijoin 
tables with tables whose records were stored in join buffers.
This fixes would become mostly unnecessary if the new code of
mwl 90 was merged into 5.3 right now.
Yet the fix the code of optimize_wo_join_buffering was needed
in any case.

- Make optimize_wo_join_buffering() handle cases where position->records_read=0 (this
  happens for outer joins that have constant tables inside them). The number of
  0 is not correct (should be 1 because outer join will produce at least a NULL-complemented
  record) but for now we just make it work with incorrect number.

The fixes for #643424 was part of the fix for #652727, that's why both
fixes are pushed together.

- The cause for #643424 was the improper use of get_partial_join_cost(),
  which assumed that the 'n_tables' parameter was the upper bound for
  query plan node indexes.
  Fixed by generalizing get_partial_join_cost() as a method that computes
  the cost of any partial join.

- The cause of #652727 was that JOIN::choose_subquery_plan() incorrectly
  deleted the contents of the old keyuse array in the cases when an injected
  plan would not provide more key accesses, and reoptimization was not actually
  performed.

- A certain problem with NULLs filtering was solved in two different 
  ways. Use MWL#128's approach (it's more generic) and rollback DS-MRR's.

Prohibited to use hash join algorithm BNLH if join attributes
need non-binary collations. It has to be done because BNLH does
not support join for such attributes yet.
Later this limitations will be lifted.

Changed default collations for the schemes of some test cases
to preserve the old execution plans.

Lines below which were added in the patch for Bug#56814 cause this crash:

+      if (table->table)
+        table->table->maybe_null= FALSE;

Consider following test case:
--
CREATE TABLE t1(f1 INT NOT NULL);
INSERT INTO t1 VALUES (16777214),(0);

SELECT COUNT(*) FROM t1 LEFT JOIN t1 t2
ON 1 WHERE t2.f1 > 1 GROUP BY t2.f1;

DROP TABLE t1;
--

We set TABLE::maybe_null to FALSE for t2 table
and in create_tmp_field() we create appropriate tmp table field
using create_tmp_field_from_item() function instead of
create_tmp_field_from_field. As a result we have
LONGLONG field. As we have GROUP BY clause we calculate
group buffer length, see calc_group_buffer().
Item from group list which is used for calculation
refer to the field from real tables and have LONG type.
So group buffer length become insufficient for storing of
LONGLONG value. It leads to overwriting of wrong memory
area in do_field_int() function which is called from
end_update().
After some investigation I found out that
create_tmp_field_from_item() is used only for OLAP
grouping and can not be used for common grouping
as it could be an incompatibility between tmp
table fields and group buffer length.
We can not remove create_tmp_field_from_item() call from
create_tmp_field as OLAP needs it and we can not use this
function for common grouping. So we should remove setting
TABLE::maybe_null to FALSE from simplify_joins().
In this case we'll get wrong behaviour of
list_contains_unique_index() back. To fix it we
could use Field::real_maybe_null() check instead of
Field::maybe_null() and add addition check of
TABLE_LIST::outer_join.


mysql-test/r/group_by.result:
  test case
mysql-test/r/join_outer.result:
  test case
mysql-test/t/group_by.test:
  test case
mysql-test/t/join_outer.test:
  test case
sql/sql_select.cc:
  --remove wrong code
  --use Field::real_maybe_null() check instead of
    Field::maybe_null() and add addition check of
    TABLE_LIST::outer_join

Post-review fix - avoid re-evaluation of the having clause
when it evaluates to true.

When join buffers are employed no index scan for the first
table with grouping columns can be used.



mysql-test/r/join_cache.result:
  Added a test case for bug #664508.
  Sorted results for some other test cases.
mysql-test/t/join_cache.test:
  Added a test case for bug #664508.
  Sorted results for some other test cases.

Added missing logic to handle the case when subquery tables are optimized
away early during optimization.

- Added more tests to the MWL#89 specific test, and made the test more modular.
- Updated test files.
- Fixed a memory leak.
- More comments.

mysql-test/r/subselect_mat.result:
  - Updated the test file to reflect the new optimizer switches related to
    materialized subquery execution.
  - Added one extra test to test all cases that expose BUG#40037 (this is an old bug from 5.x).
  - Updated the test result with correct results that expose BUG#40037.
mysql-test/t/subselect_mat.test:
  - Updated the test file to reflect the new optimizer switches related to
    materialized subquery execution.
  - Added one extra test to test all cases that expose BUG#40037 (this is an old bug from 5.x).
  - Updated the test result with correct results that expose BUG#40037.
sql/sql_select.cc:
  Fixed a memory leak reported by Valgrind.