Bug #18463911 : SERVER CRASHES ON CREATING A TEMP TABLE

                WITH CERTAIN MAX_HEAP_TABLE_SIZE VALUES

Description:
When the  system variable 'max_heap_table_size'
is set to 20GB, the server crashes on creation of a
temporary tables or tables using MEMORY storage engine.

Analysis:
The variable 'max_record' determines the amount heap
allocated for the records of the table. This value
is determined using the 'max_heap_table_size' variable.
'records_in_block' in turn uses the max_records to
determine the number of records per block.

When the 'max_heap_table_size' is set to 20GB, then
the 'records_in_block' is calculated to a value of
2^28.

The size of the block determined by multiplying the
'records_in_block' and 'recbuffer' results in overflow
and hence the value becomes zero. As a result, zero bytes
of the heap is allocated for the table. This will
result in a server crash when the table is accessed.

Fix:
The variables 'records_in_block' and 'recbuffer' are
typecasted to 'unsigned long' while calculating the
size of the block.

storage/heap/hp_block.c

-/* Copyright (c) 2000, 2011, Oracle and/or its affiliates. All rights reserved.
+/* Copyright (c) 2000, 2014, Oracle and/or its affiliates. All rights reserved.
 
    This program is free software; you can redistribute it and/or modify
    it under the terms of the GNU General Public License as published by
@@ -75,7 +75,8 @@ int hp_get_new_block(HP_BLOCK *block, size_t *alloc_length)
     This doesn't add much overhead - with current values of sizeof(HP_PTRS) 
     and my_default_record_cache_size we get about 1/128 unused memory.
    */
-  *alloc_length=sizeof(HP_PTRS)*i+block->records_in_block* block->recbuffer;
+  *alloc_length= sizeof(HP_PTRS)* i + (ulong) block->records_in_block *
+                                              block->recbuffer;
   if (!(root=(HP_PTRS*) my_malloc(*alloc_length,MYF(MY_WME))))
     return 1;
 

storage/heap/hp_create.c

-/* Copyright (c) 2000, 2011, Oracle and/or its affiliates. All rights reserved.
+/* Copyright (c) 2000, 2014, Oracle and/or its affiliates. All rights reserved.
 
    This program is free software; you can redistribute it and/or modify
    it under the terms of the GNU General Public License as published by
@@ -240,7 +240,7 @@ static void init_block(HP_BLOCK *block, uint reclength, ulong min_records,
   records_in_block= max_records / 10;
   if (records_in_block < 10 && max_records)
     records_in_block= 10;
-  if (!records_in_block || records_in_block*recbuffer >
+  if (!records_in_block || (ulong) records_in_block * recbuffer >
       (my_default_record_cache_size-sizeof(HP_PTRS)*HP_MAX_LEVELS))
     records_in_block= (my_default_record_cache_size - sizeof(HP_PTRS) *
 		      HP_MAX_LEVELS) / recbuffer + 1;