BUG#11392 - fulltext search bug

Fulltext boolean mode phrase search may crash server on platforms
where size of pointer is not equal to size of unsigned integer
(in other words some 64-bit platforms).

The problem was integer overflow.

Affects 4.1 only.


myisam/ft_boolean_search.c:
  my_match_t::beg is unsigned int, that means type of expression
  (m[0].beg - 1) has unsigned type too. It may happen that instr()
  finds substring in the beggining of passed string, returning
  m[0].beg equal to 0. In this case value of expression (m[0].beg - 1)
  is equal to MAX_UINT.
  
  This is not a problem on platforms where sizeof(pointer) equals to
  sizeof(uint). That means ptr[(uint)-1] = ptr[(uint)MAX_UINT] = ptr - 1.
  
  On some 64-bit platforms where sizeof(pointer) is 8 and sizeof(uint)
  is 4, wrong address gets accessed. In other words ptr[(uint)-1] is
  equal to ptr + MAX_UINT.
mysql-test/r/fulltext.result:
  A test case for BUG#11392.
mysql-test/t/fulltext.test:
  A test case for BUG#11392.

myisam/ft_boolean_search.c

@@ -446,7 +446,8 @@ static int _ftb_strstr(const byte *s0, const byte *e0,
   {
     if (cs->coll->instr(cs, p0, e0 - p0, s1, e1 - s1, m, 2) != 2)
       return(0);
-    if ((!s_after || p0 + m[1].beg == s0 || !true_word_char(cs, p0[m[1].beg-1])) &&
+    if ((!s_after || p0 + m[1].beg == s0 ||
+         !true_word_char(cs, p0[(int) m[1].beg - 1])) &&
         (!e_before || p0 + m[1].end == e0 || !true_word_char(cs, p0[m[1].end])))
       return(1);
     p0+= m[1].beg;

mysql-test/r/fulltext.result

@@ -454,3 +454,9 @@ ALTER TABLE t1 DISABLE KEYS;
 SELECT * FROM t1 WHERE MATCH(a) AGAINST('test');
 ERROR HY000: Can't find FULLTEXT index matching the column list
 DROP TABLE t1;
+CREATE TABLE t1(a TEXT);
+INSERT INTO t1 VALUES(' aaaaa aaaa');
+SELECT * FROM t1 WHERE MATCH(a) AGAINST ('"aaaa"' IN BOOLEAN MODE);
+a
+ aaaaa aaaa
+DROP TABLE t1;

mysql-test/t/fulltext.test

@@ -379,4 +379,12 @@ ALTER TABLE t1 DISABLE KEYS;
 SELECT * FROM t1 WHERE MATCH(a) AGAINST('test');
 DROP TABLE t1;
 
+#
+# BUG#11392 - fulltext search bug
+#
+CREATE TABLE t1(a TEXT);
+INSERT INTO t1 VALUES(' aaaaa aaaa');
+SELECT * FROM t1 WHERE MATCH(a) AGAINST ('"aaaa"' IN BOOLEAN MODE);
+DROP TABLE t1;
+
 # End of 4.1 tests