The problem was introduced by 1859caf6:
  MDEV-10175: range optimizer calls records_in_range() for full extended keys

  Make the range optimizer not call records_in_range() when it would
  not give any benefit.

that patch used an incorrect way to check for full extended key. Now fixing
the check.