The source of the problem is in Field_longlong::cmp. If 'this' is
  an unsigned number, the method casts both the current value, and
  the constant that we compare with to an unsigned number. As a
  result if the constant we compare with is a negative number, it
  wraps to some unsigned number, and the comparison is incorrect.
  
  When the optimizer chooses the "range" access method, this problem
  causes handler::read_range_next to reject the current key when the
  upper bound key is a negative number because handler::compare_key
  incorrectly considers the positive and negative keys to be equal.
  
  The current patch does not correct the source of the problem in
  Field_longlong::cmp because it is not easy to propagate sign
  information about the constant at query execution time. Instead
  the patch changes the range optimizer so that it never compares
  unsiged fields with negative constants. As an added benefit,
  queries that do such comparisons will execute faster because
  the range optimizer replaces conditions like:
  (a) (unsigned_int [< | <=] negative_constant) == FALSE
  (b) (unsigned_int [> | >=] negative_constant) == TRUE
  with the corresponding constants.
  In some cases this may even result in constant time execution.


mysql-test/r/range.result:
  - Added test for BUG#11185
  - Added missing test from 4.1. This test also tests the fix for BUG#11185.
mysql-test/t/range.test:
  - Added test for BUG#11185
  - Added missing test from 4.1. This test also tests the fix for BUG#11185.
sql/opt_range.cc:
  Added a new optimization to the range optimizer where we detect that
  an UNSIGNED field is compared with a negative constant. Depending on
  the comparison operator, we know directly that the result of the
  comparison is either TRUE or FALSE for all input values, and we need
  not check each value.
      
  This optimization is also necessary so that the index range access
  method produces correct results when comparing unsigned fields with
  negative constants.