build)

The crash was caused by freeing the internal parser stack during the parser
execution.
This occured only for complex stored procedures, after reallocating the parser
stack using my_yyoverflow(), with the following C call stack:
- MYSQLparse()
- any rule calling sp_head::restore_lex()
- lex_end()
- x_free(lex->yacc_yyss), xfree(lex->yacc_yyvs)

The root cause is the implementation of stored procedures, which breaks the
assumption from 4.1 that there is only one LEX structure per parser call.

The solution is to separate the LEX structure into:
- attributes that represent a statement (the current LEX structure),
- attributes that relate to the syntax parser itself (Yacc_state),
so that parsing multiple statements in stored programs can create multiple
LEX structures while not changing the unique Yacc_state.

Now, Yacc_state and the existing Lex_input_stream are aggregated into
Parser_state, a structure that represent the complete state of the (Lexical +
Syntax) parser.

Mixing aggregate functions and non-grouping columns is not allowed in the
ONLY_FULL_GROUP_BY mode. However in some cases the error wasn't thrown because
of insufficient check.

In order to check more thoroughly the new algorithm employs a list of outer
fields used in a sum function and a SELECT_LEX::full_group_by_flag.
Each non-outer field checked to find out whether it's aggregated or not and
the current select is marked accordingly.
All outer fields that are used under an aggregate function are added to the
Item_sum::outer_fields list and later checked by the Item_sum::check_sum_func
function.

When the server was out of memory it crashed because of invalid memory access.

This patch adds detection for failed memory allocations and make the server
output a proper error message.

Currently the Last_query_cost session status variable shows
only the cost of a single flat subselect. For complex queries
(with subselects or unions etc) Last_query_cost is not valid
as it was showing the cost for the last optimized subselect.
Fixed by reseting to zero Last_query_cost when the complete
cost of the query cannot be determined.
Last_query_cost will be non-zero only for single flat queries.

The bug caused memory corruption for some queries with top OR level
in the WHERE condition if they contained equality predicates and 
other sargable predicates in disjunctive parts of the condition.

The corruption happened because the upper bound of the memory
allocated for KEY_FIELD and SARGABLE_PARAM internal structures
containing info about potential lookup keys was calculated incorrectly
in some cases. In particular it was calculated incorrectly when the
WHERE condition was an OR formula with disjuncts being AND formulas
including equalities and other sargable predicates.

(Regression, caused by a patch for the bug 22646).
Problem: when result type of date_format() was changed from
binary string to character string, mixing date_format()
with a ascii column in CONCAT() stopped to work.
Fix:
- adding "repertoire" flag into DTCollation class,
to mark items which can return only pure ASCII strings.
- allow character set conversion from pure ASCII to other character sets.

causes full table lock on innodb table.
Also fixes Bug#28502 Triggers that update another innodb table 
will block on X lock unnecessarily (duplciate).
Code review fixes.

Both bugs' synopses are misleading: InnoDB table is
not X locked. The statements, however, cannot proceed concurrently, 
but this happens due to lock conflicts for tables used in triggers,
not for the InnoDB table. 

If a user had an InnoDB table, and two triggers, AFTER UPDATE and 
AFTER INSERT, competing for different resources (e.g. two distinct
MyISAM tables), then these two triggers would not be able to execute
concurrently. Moreover, INSERTS/UPDATES of the InnoDB table would
not be able to run concurrently. 
The problem had other side-effects (see respective bug reports).

This behavior was a consequence of a shortcoming of the pre-locking
algorithm, which would not distinguish between different DML operations
(e.g. INSERT and DELETE) and pre-lock all the tables
that are used by any trigger defined on the subject table.

The idea of the fix is to extend the pre-locking algorithm to keep track,
for each table, what DML operation it is used for and not
load triggers that are known to never be fired.

The need arose when working on Bug 26141, where it became
necessary to replace TABLE_LIST with its forward declaration in a few
headers, and this involved a lot of s/TABLE_LIST/st_table_list/.
Although other workarounds exist, this patch is in line
with our general strategy of moving away from typedef-ed names.
Sometime in future we might also rename TABLE_LIST to follow the
coding style, but this is a huge change.

fails if a database is not selected prior.

The problem manifested itself when a user tried to
create a routine that had non-fully-qualified identifiers in its bodies
and there was no current database selected.

This is a regression introduced by the fix for Bug 19022:

The patch for Bug 19022 changes the code to always produce a warning
if we can't resolve the current database in the parser. 
In this case this was not necessary, since even though the produced
parsed tree was incorrect, we never re-use sphead
that was obtained at first parsing of CREATE PROCEDURE.
The sphead that is anyhow used is always obtained through db_load_routine,
and there we change the current database to sphead->m_db before
calling yyparse.

The idea of the fix is to resolve the current database directly using 
lex->sphead->m_db member when parsing a stored routine body, when
such is present.

This patch removes the need to reset the current database
when loading a trigger or routine definition into SP cache.
The redundant code will be removed in 5.1.

The root cause of this bug is related to the function skip_rear_comments,
in sql_lex.cc

Recent code changes in skip_rear_comments changed the prototype from
"const uchar*" to "const char*", which had an unforseen impact on this test:
  (endp[-1] < ' ')
With unsigned characters, this code filters bytes of value [0x00 - 0x20]
With *signed* characters, this also filters bytes of value [0x80 - 0xFF].

This caused the regression reported, considering cyrillic characters in the
parameter name to be whitespace, and truncated.
Note that the regression is present both in 5.0 and 5.1.

With this fix:
- [0x80 - 0xFF] bytes are no longer considered whitespace.
This alone fixes the regression.

In addition, filtering [0x00 - 0x20] was found bogus and abusive,
so that the code now filters uses my_isspace when looking for whitespace.

Note that this fix is only addressing the regression affecting UTF-8
in general, but does not address a more fundamental problem with
skip_rear_comments: parsing a string *backwards*, starting at end[-1],
is not safe with multi-bytes characters, so that end[-1] can confuse the
last byte of a multi-byte characters with a characters to filter out.

The only known impact of this remaining issue affects objects that have to
meet all the conditions below:

- the object is a FUNCTION / PROCEDURE / TRIGGER / EVENT / VIEW
- the body consist of only *1* instruction, and does *not* contain a
  BEGIN-END block
- the instruction ends, lexically, with <ident> <whitespace>* ';'?
  For example, "select <ident>;" or "return <ident>;"
- The last character of <ident> is a multi-byte character
- the last byte of this character is ';' '*', '/' or whitespace

In this case, the body of the object will be truncated after parsing,
and stored in an invalid format.

This last issue has not been fixed in this patch, since the real fix
will be implemented by Bug 25411 (trigger code truncated), which is caused
by the very same code.
The real problem is that the function skip_rear_comments is only a
work-around, and should be removed entirely: see the proposed patch for
bug 25411 for details.

Bug#21483 "Server abort or deadlock on INSERT DELAYED with another
implicit insert"
Also fixes and adds test cases for bugs:
20497 "Trigger with INSERT DELAYED causes Error 1165"
21714 "Wrong NEW.value and server abort on INSERT DELAYED to a
table with a trigger".
Post-review fixes.

Problem:
In MySQL INSERT DELAYED is a way to pipe all inserts into a
given table through a dedicated thread. This is necessary for
simplistic storage engines like MyISAM, which do not have internal
concurrency control or threading and thus can not
achieve efficient INSERT throughput without support from SQL layer.
DELAYED INSERT works as follows:
For every distinct table, which can accept DELAYED inserts and has
pending data to insert, a dedicated thread is created to write data
to disk. All user connection threads that attempt to
delayed-insert into this table interact with the dedicated thread in
producer/consumer fashion: all records to-be inserted are pushed
into a queue of the dedicated thread, which fetches the records and 
writes them.
In this design, client connection threads never open or lock
the delayed insert table.
This functionality was introduced in version 3.23 and does not take 
into account existence of triggers, views, or pre-locking.
E.g. if INSERT DELAYED is called from a stored function, which,
in turn, is called from another stored function that uses the delayed
table, a deadlock can occur, because delayed locking by-passes
pre-locking. Besides:
 * the delayed thread works directly with the subject table through
   the storage engine API and does not invoke triggers
 * even if it was patched to invoke triggers, if triggers,
   in turn, used other tables, the delayed thread would
   have to open and lock involved tables (use pre-locking).
 * even if it was patched to use pre-locking, without deadlock
   detection the delayed thread could easily lock out user 
   connection threads in case when the same table is used both
   in a trigger and on the right side of the insert query: 
   the delayed thread would not release locks until all inserts 
   are complete, and user connection can not complete inserts 
   without having locks on the tables used on the right side of the
   query.

Solution:

These considerations suggest two general alternatives for the
future of INSERT DELAYED:
 * it is considered a full-fledged alternative to normal INSERT
 * it is regarded as an optimisation that is only relevant 
   for simplistic engines.
Since we missed our chance to provide complete support of new
features when 5.0 was in development, the first alternative
currently renders infeasible.
However, even the second alternative, which is to detect
new features and convert DELAYED insert into a normal insert, 
is not easy to implement.
The catch-22 is that we don't know if the subject table has triggers
or is a view before we open it, and we only open it in the
delayed thread. We don't know if the query involves pre-locking
until we have opened all tables, and we always first create
the delayed thread, and only then open the remaining tables.
This patch detects the problematic scenarios and converts
DELAYED INSERT to a normal INSERT using the following approach:
 * if the statement is executed under pre-locking (e.g. from
   within a stored function or trigger) or the right
   side may require pre-locking, we detect the situation
   before creating a delayed insert thread and convert the statement
   to a conventional INSERT.
  * if the subject table is a view or has triggers, we shutdown
   the delayed thread and convert the statement to a conventional
   INSERT.

by moving yet another relevant flag to it from struct LEX.

The issue found with bug 25411 is due to the function skip_rear_comments()
which damages the source code while implementing a work around.
The root cause of the problem is in the lexical analyser, which does not
process special comments properly.
For special comments like :
[1] aaa /*!50000 bbb */ ccc
since 5.0 is a version older that the current code, the parser is in lining
the content of the special comment, so that the query to process is
[2] aaa bbb ccc
However, the text of the query captured when processing a stored procedure,
stored function or trigger (or event in 5.1), can be after rebuilding it:
[3] aaa bbb */ ccc
which is wrong.

To fix bug 25411 properly, the lexical analyser needs to return [2] when
in lining special comments.
In order to implement this, some preliminary cleanup is required in the code,
which is implemented by this patch.

Before this change, the structure named LEX (or st_lex) contains attributes
that belong to lexical analysis, as well as attributes that represents the
abstract syntax tree (AST) of a statement.
Creating a new LEX structure for each statements (which makes sense for the
AST part) also re-initialized the lexical analysis phase each time, which
is conceptually wrong.

With this patch, the previous st_lex structure has been split in two:
- st_lex represents the Abstract Syntax Tree for a statement. The name "lex"
has not been changed to avoid a bigger impact in the code base.
- class lex_input_stream represents the internal state of the lexical
  analyser, which by definition should *not* be reinitialized when parsing
  multiple statements from the same input stream.

This change is a pre-requisite for bug 25411, since the implementation of
lex_input_stream will later improve to deal properly with special comments,
and this processing can not be done with the current implementation of
sp_head::reset_lex and sp_head::restore_lex, which interfere with the lexer.

This change set alone does not fix bug 25411.

When merging views into the enclosing statement
the ORDER BY clause of the view is merged to the
parent's ORDER BY clause.
However when the VIEW is merged into an UNION
branch the ORDER BY should be ignored. 
Use of ORDER BY for individual SELECT statements
implies nothing about the order in which the rows
appear in the final result because UNION by default
produces unordered set of rows.
Fixed by ignoring the ORDER BY clause from the merge
view when expanded in an UNION branch.

away.

During optimization stage the WHERE conditions can be changed or even
be removed at all if they know for sure to be true of false. Thus they aren't
showed in the EXPLAIN EXTENDED which prints conditions after optimization.

Now if all elements of an Item_cond were removed this Item_cond is substituted
for an Item_int with the int value of the Item_cond.
If there were conditions that were totally optimized away then values of the
saved cond_value and having_value will be printed instead.

fixes).

The legend: on a replication slave, in case a trigger creation
was filtered out because of application of replicate-do-table/
replicate-ignore-table rule, the parsed definition of a trigger was not 
cleaned up properly. LEX::sphead member was left around and leaked 
memory. Until the actual implementation of support of 
replicate-ignore-table rules for triggers by the patch for Bug 24478 it 
was never the case that "case SQLCOM_CREATE_TRIGGER"
was not executed once a trigger was parsed,
so the deletion of lex->sphead there worked and the memory did not leak.

The fix: 

The real cause of the bug is that there is no 1 or 2 places where
we can clean up the main LEX after parse. And the reason we 
can not have just one or two places where we clean up the LEX is
asymmetric behaviour of MYSQLparse in case of success or error. 

One of the root causes of this behaviour is the code in Item::Item()
constructor. There, a newly created item adds itself to THD::free_list
- a single-linked list of Items used in a statement. Yuck. This code
is unaware that we may have more than one statement active at a time,
and always assumes that the free_list of the current statement is
located in THD::free_list. One day we need to be able to explicitly
allocate an item in a given Query_arena.
Thus, when parsing a definition of a stored procedure, like
CREATE PROCEDURE p1() BEGIN SELECT a FROM t1; SELECT b FROM t1; END;
we actually need to reset THD::mem_root, THD::free_list and THD::lex
to parse the nested procedure statement (SELECT *).
The actual reset and restore is implemented in semantic actions
attached to sp_proc_stmt grammar rule.
The problem is that in case of a parsing error inside a nested statement
Bison generated parser would abort immediately, without executing the
restore part of the semantic action. This would leave THD in an 
in-the-middle-of-parsing state.
This is why we couldn't have had a single place where we clean up the LEX
after MYSQLparse - in case of an error we needed to do a clean up
immediately, in case of success a clean up could have been delayed.
This left the door open for a memory leak.

One of the following possibilities were considered when working on a fix:
- patch the replication logic to do the clean up. Rejected
as breaks module borders, replication code should not need to know the
gory details of clean up procedure after CREATE TRIGGER.
- wrap MYSQLparse with a function that would do a clean up.
Rejected as ideally we should fix the problem when it happens, not
adjust for it outside of the problematic code.
- make sure MYSQLparse cleans up after itself by invoking the clean up
functionality in the appropriate places before return. Implemented in 
this patch.
- use %destructor rule for sp_proc_stmt to restore THD - cleaner
than the prevoius approach, but rejected
because needs a careful analysis of the side effects, and this patch is 
for 5.0, and long term we need to use the next alternative anyway
- make sure that sp_proc_stmt doesn't juggle with THD - this is a 
large work that will affect many modules.

Cleanup: move main_lex and main_mem_root from Statement to its
only two descendants Prepared_statement and THD. This ensures that
when a Statement instance was created for purposes of statement backup,
we do not involve LEX constructor/destructor, which is fairly expensive.
In order to track that the transformation produces equivalent 
functionality please check the respective constructors and destructors
of Statement, Prepared_statement and THD - these members were
used only there.
This cleanup is unrelated to the patch.

  Post fix for bug#23800.
  The Item_field constructor now increases the select_n_where_fields counter.
sql_yacc.yy:
  Post fix for bug#23800.
  Take into account fields that might be added by subselects.
sql_lex.h:
  Post fix for bug#23800.
  Added the select_n_where_fields variable to the st_select_lex class.
sql_lex.cc:
  Post fix for bug#23800.
  Initialization of the select_n_where_fields variable.

created for sorting.

Any outer reference in a subquery was represented by an Item_field object.
If the outer select employs a temporary table all such fields should be
replaced with fields from that temporary table in order to point to the 
actual data. This replacement wasn't done and that resulted in a wrong
subquery evaluation and a wrong result of the whole query.

Now any outer field is represented by two objects - Item_field placed in the
outer select and Item_outer_ref in the subquery. Item_field object is
processed as a normal field and the reference to it is saved in the
ref_pointer_array. Thus the Item_outer_ref is always references the correct
field. The original field is substituted for a reference in the
Item_field::fix_outer_field() function.

New function called fix_inner_refs() is added to fix fields referenced from
inner selects and to fix references (Item_ref objects) to these fields.

The new Item_outer_ref class is a descendant of the Item_direct_ref class.
It additionally stores a reference to the original field and designed to
behave more like a field.

           fails

The bug was introduced with the push of the fix for bug#20953: after
the error on view creation we never reset the error state, so some
valid statements would give the same error after that.

The solution is to properly reset the error state.

 Two problems here:

 Problem 1:

 While constructing the join columns list the optimizer does as follows:
  1. Sets the join_using_fields/natural_join members of the right JOIN 
   operand.
  2. Makes a "table reference" (TABLE_LIST) to parent the two tables.
  3. Assigns the join_using_fields/is_natural_join of the wrapper table
   using join_using_fields/natural_join of the rightmost table
  4. Sets join_using_fields to NULL for the right JOIN operand.
  5. Passes the parent table up to the same procedure on the upper 
   level.

 Step 1 overrides the the join_using_fields that are set for a nested 
 join wrapping table in step 4.
 Fixed by making a designated variable SELECT_LEX::prev_join_using to 
 pass the data from step 1 to step 4 without destroying the wrapping 
 table data.

 Problem 2:

 The optimizer checks for ambiguous columns while transforming 
 NATURAL JOIN/JOIN USING to JOIN ON. While doing that there was no
 distinction between columns that are used in the generated join
 condition (where ambiguity can be checked) and the other columns
 (where ambiguity can be checked only when resolving references
 coming from outside the JOIN construct itself).
 Fixed by allowing the non-USING columns to be present in multiple 
 copies in both sides of the join and moving the ambiguity check 
 to the place where unqualified references to the join columns are
 resolved (find_field_in_natural_join()).

- Make the code produce correct result: use an array of triggers to turn on/off equalities for each
  compared column. Also turn on/off optimizations based on those equalities.
- Make EXPLAIN output show "Full scan on NULL key" for tables for which we switch between
  ref/unique_subquery/index_subquery and ALL access.
- index_subquery engine now has HAVING clause when it is needed, and it is
  displayed in EXPLAIN EXTENDED
- Fix incorrect presense of "Using index" for index/unique-based subqueries (BUG#22930)
// bk trigger note: this commit refers to BUG#24127

Currently in the ONLY_FULL_GROUP_BY mode no hidden fields are allowed in the
select list. To ensure this each expression in the select list is checked
to be a constant, an aggregate function or to occur in the GROUP BY list.
The last two requirements are wrong and doesn't allow valid expressions like
"MAX(b) - MIN(b)" or "a + 1" in a query with grouping by a.

The correct check implemented by the patch will ensure that:
any field reference in the [sub]expressions of the select list 
  is under an aggregate function or
  is mentioned as member of the group list or
  is an outer reference or
  is part of the select list element that coincide with a grouping element.

The Item_field objects now can contain the position of the select list
expression which they belong to. The position is saved during the
field's Item_field::fix_fields() call.

The non_agg_fields list for non-aggregated fields is added to the SELECT_LEX
class. The SELECT_LEX::cur_pos_in_select_list now contains the position in the
select list of the expression being currently fixed.

  Corrected spelling in copyright text
Makefile.am:
  Don't update the files from BitKeeper
Many files:
  Removed "MySQL Finland AB & TCX DataKonsult AB" from copyright header
  Adjusted year(s) in copyright header 
Many files:
  Added GPL copyright text
Removed files:
  Docs/Support/colspec-fix.pl
  Docs/Support/docbook-fixup.pl
  Docs/Support/docbook-prefix.pl
  Docs/Support/docbook-split
  Docs/Support/make-docbook
  Docs/Support/make-makefile
  Docs/Support/test-make-manual
  Docs/Support/test-make-manual-de
  Docs/Support/xwf

  Changed header to GPL version 2 only

- Removed not used variables and functions
- Added #ifdef around code that is not used
- Renamed variables and functions to avoid conflicts
- Removed some not used arguments

Fixed some class/struct warnings in ndb
Added define IS_LONGDATA() to simplify code in libmysql.c

I did run gcov on the changes and added 'purecov' comments on almost all lines that was not just variable name changes

Bug#4968 "Stored procedure crash if cursor opened on altered table"
Bug#19733 "Repeated alter, or repeated create/drop, fails"
Bug#19182 "CREATE TABLE bar (m INT) SELECT n FROM foo; doesn't work from 
stored procedure."
Bug#6895 "Prepared Statements: ALTER TABLE DROP COLUMN does nothing"
Bug#22060 "ALTER TABLE x AUTO_INCREMENT=y in SP crashes server"

Test cases for bugs 4968, 19733, 6895 will be added in 5.0.

Re-execution of CREATE DATABASE, CREATE TABLE and ALTER TABLE 
statements in stored routines or as prepared statements caused
incorrect results (and crashes in versions prior to 5.0.25).
In 5.1 the problem occured only for CREATE DATABASE, CREATE TABLE
SELECT and CREATE TABLE with INDEX/DATA DIRECTOY options).

The problem of bugs 4968, 19733, 19282 and 6895 was that functions
mysql_prepare_table, mysql_create_table and mysql_alter_table were not
re-execution friendly: during their operation they used to modify contents
of LEX (members create_info, alter_info, key_list, create_list),
thus making the LEX unusable for the next execution.
In particular, these functions removed processed columns and keys from
create_list, key_list and drop_list. Search the code in sql_table.cc 
for drop_it.remove() and similar patterns to find evidence.

The fix is to supply to these functions a usable copy of each of the
above structures at every re-execution of an SQL statement. 

To simplify memory management, LEX::key_list and LEX::create_list
were added to LEX::alter_info, a fresh copy of which is created for
every execution.

The problem of crashing bug 22060 stemmed from the fact that the above 
metnioned functions were not only modifying HA_CREATE_INFO structure in 
LEX, but also were changing it to point to areas in volatile memory of 
the execution memory root.
 
The patch solves this problem by creating and using an on-stack
copy of HA_CREATE_INFO (note that code in 5.1 already creates and
uses a copy of this structure in mysql_create_table()/alter_table(),
but this approach didn't work well for CREATE TABLE SELECT statement).

Fixed compiler warnings (detected by VC++):
- Removed not used variables
- Added casts
- Fixed wrong assignments to bool
- Fixed wrong calls with bool arguments
- Added missing argument to store(longlong), which caused wrong store method to be called.

  limitation)

Note to the reviewer
====================

Warning: reviewing this patch is somewhat involved.
Due to the nature of several issues all affecting the same area,
fixing separately each issue is not practical, since each fix can not be
implemented and tested independently.
In particular, the issues with
- rule recursion
- nested case statements
- forward jump resolution (backpatch list)
are tightly coupled (see below).

Definitions
===========

The expression
  CASE expr
  WHEN expr THEN expr
  WHEN expr THEN expr
  ...
  END
is a "Simple Case Expression".

The expression
  CASE
  WHEN expr THEN expr
  WHEN expr THEN expr
  ...
  END
is a "Searched Case Expression".

The statement
  CASE expr
  WHEN expr THEN stmts
  WHEN expr THEN stmts
  ...
  END CASE
is a "Simple Case Statement".

The statement
  CASE
  WHEN expr THEN stmts
  WHEN expr THEN stmts
  ...
  END CASE
is a "Searched Case Statement".

A "Left Recursive" rule is like
  list:
      element
    | list element
    ;

A "Right Recursive" rule is like
  list:
      element
    | element list
    ;

Left and right recursion produces the same language, the difference only
affects the *order* in which the text is parsed.

In a descendant parser (usually written manually), right recursion works
very well, and is typically implemented with a while loop.
In an ascendant parser (yacc/bison) left recursion works very well,
and is implemented naturally by the parser stack.
In both cases, using the wrong type or recursion is very bad and should be
avoided, as it causes technical issues with the parser implementation.

Before this change
==================

The "Simple Case Expression" and "Searched Case Expression" were both
implemented by the "when_list" and "when_list2" rules, which are left
recursive (ok).

These rules, however, used lex->when_list instead of using the parser stack,
which is more complex that necessary, and potentially dangerous because
of other rules using THD::reset_lex.

The "Simple Case Statement" and "Searched Case Statements" were implemented
by the "sp_case", "sp_whens" and in part by "sp_proc_stmt" rules.
Both cases were right recursive (bad).

The grammar involved was convoluted, and is assumed to be the results of
tweaks to get the code generation to work, but is not what someone would
naturally write.

In addition, using a common rule for both "Simple" and "Searched" case
statements was implemented with sp_head::m_flags |= IN_SIMPLE_CASE,
which is a flag and not a stack, and therefore does not take into account
*nested* case statements. This leads to incorrect generated code, and either
a server crash or an incorrect result.

With regards to the backpatch mechanism, a *different* backpatch list was
created for each jump from "WHEN expr THEN stmt" to "END CASE", which
relied on the grammar to be right recursive.
This is a mis-use of the backpatch list, since this list can resolve
multiple references to the same target at once.

The optimizer algorithm used to detect dead code in the "assembly" SQL
instructions, implemented by sp_head::opt_mark(uint ip), was recursive
in some cases (a conditional jump pointing forward to another conditional
jump).
In case of specially crafted code, like
- a long list of "IF expr THEN stmt END IF"
- a long CASE statement
this would actually cause a server crash with a stack overflow.
In general, having a stack that grows proportionally with user data (the
SQL code given by the client in a CREATE PROCEDURE) is to be avoided.

In debug builds only, creating a SP / SF / Trigger which had a significant
amount of code would spend --literally-- several minutes in sp_head::create,
because of the debug code involved with DBUG_PRINT("info", ("Code %s ...
There are several issues with this code:
- in a CASE with 5 000 WHEN, there are 15 000 instructions generated,
  which create a sting representation of the code which is 500 000 bytes
  long,
- using a String instead of an io stream causes performances to degrade
  to a total server freeze, as time is spent doing realloc of a buffer
  always too short,
- Printing a 500 000 long string in the debug log is too verbose,
- Generating this string even when DBUG_PRINT is off is useless,
- Having code that potentially can affect the server behavior, used with
  #ifdef / #endif is useful in some cases, but is also a bad practice.

After this change
=================

"Case Expressions" (both simple and searched) have been simplified to
not use LEX::when_list, which has been removed.

Considering all the issues affecting case statements, the grammar for these
has been totally re written.

The existing actions, used to generate "assembly" sp_inst* code, have been
preserved but moved in the new grammar, with the following changes:

a) Bison rules are no longer shared between "Simple" and "Searched" case
statements, because a stack instead of a flag is required to handle them.
Nested statements are handled naturally by the parser stack, which by
definition uses the correct rule in the correct context.
Nested statements of the opposite type (simple vs searched) works correctly.
The flag sp_head::IN_SIMPLE_CASE is no longer used.
This is a step towards resolution of WL#2999, which correctly identified
that temporary parsing flags do not belong to sp_head.
The code in the action is shared by mean of the case_stmt_action_xxx()
helpers.

b) The backpatch mechanism, used to resolve forward jumps in the generated
code, has been changed to:
- create a label for the instruction following 'END CASE',
- register each jump at the end of a "WHEN expr THEN stmt" in a *unique*
  backpatch list associated with the 'END CASE' label
- resolve all the forward jumps for this label at once.

In addition, the code involving backpatch has been commented, so that a
reader can now understand by reading matching "Registering" and "Resolving"
comments how the forward jumps are resolved and what target they resolve to,
as this is far from evident when reading the code alone.

The implementation of sp_head::opt_mark() has been revised to avoid
recursive calls from jump instructions, and instead add the jump location
to the list of paths to explore during the flow analysis of the instruction
graph, with a call to sp_head::add_mark_lead().
In addition, the flow analysis will stop if an instruction has already
been marked as reachable, which the previous code failed to do in the
recursive case.
sp_head::opt_mark() is now private, to prevent new calls to this method from
being introduced.

The debug code present in sp_head::create() has been removed.
Considering that SHOW PROCEDURE CODE is also available in debug builds,
and can be used anytime regardless of the trace level, as opposed to
"CREATE PROCEDURE" time and only if the trace was on,
removing the code actually makes debugging easier (usable trace).

Tests have been written to cover the parser overflow (big CASE),
and to cover nested CASE statements.

Use lazy initialization for Query_tables_list::sroutines hash.
This step should significantly decrease amount of memory consumed
by stored routines as we no longer will allocate chunk of memory
required for this HASH for each statement in routine.

Evaluate "NULL IN (SELECT ...)" in a special way: Disable pushed-down 
conditions and their "consequences": 
 = Do full table scans instead of unique_[index_subquery] lookups.
 = Change appropriate "ref_or_null" accesses to full table scans in
   subquery's joins.
Also cache value of NULL IN (SELECT ...) if the SELECT is not correlated 
wrt any upper select.

            select OK.
The SQL parser was using Item::name to transfer user defined function attributes
to the user defined function (udf). It was not distinguishing between user defined 
function call arguments and stored procedure call arguments. Setting Item::name 
was causing Item_ref::print() method to print the argument as quoted identifiers 
and caused views that reference aggregate functions as udf call arguments (and 
rely on Item::print() for the text of the view to store) to throw an undefined 
identifier error.
Overloaded Item_ref::print to print aggregate functions as such when printing
the references to aggregate functions taken out of context by split_sum_func2()
Fixed the parser to properly detect using AS clause in stored procedure arguments
as an error.
Fixed printing the arguments of udf call to print properly the udf attribute.

account predicates that become sargable after reading const tables.
In some cases this resulted in choosing non-optimal execution plans.
Now info of such potentially saragable predicates is saved in
an array and after reading const tables we check whether this
predicates has become saragable.

           should fail to create

The problem was that this type of errors was checked during view
creation, which doesn't happen when CREATE VIEW is a statement of
a created stored routine.

The solution is to perform the checks at parse time.  The idea of the
fix is that the parser checks if a construction just parsed is allowed
in current circumstances by testing certain flags, and this flags are
reset for VIEWs.

The side effect of this change is that if the user already have
such bogus routines, it will now get a error when trying to do

  SHOW CREATE PROCEDURE proc;

(and some other) and when trying to execute such routine he will get

  ERROR 1457 (HY000): Failed to load routine test.p5. The table mysql.proc is missing, corrupt, or contains bad data (internal code -6)

However there should be very few such users (if any), and they may
(and should) drop these bogus routines.

Set a flag when a SHOW command is parsed, and check it in log_slow_statement().  SHOW commands are not counted as slow queries, even if they use table scans.

containing a select statement that uses an aggregating IN subquery.
Added a parameter to the function fix_prepare_information 
to restore correctly the having clause for the second execution.
Saved andor structure of the having conditions at the proper moment
before any calls of split_sum_func2 that could modify the having structure
adding new Item_ref objects. (These additions, are produced not with 
the statement mem_root, but rather with the execution mem_root.)

make st_select_lex::setup_ref_array() take into account that 
Item_sum-descendant objects located within descendant SELECTs
may be added into ref_pointer_array.

  - if there are two character set definitions in the column declaration,
    we replace the first one with the second one as we store both in the LEX->charset
    slot. Add a separate slot to the LEX structure to store underscore charset.
  - convert default values to the column charset of STRING, VARSTRING fields 
    if necessary as well.

  Correct memory leak fix