Upon #storage reaching #autostart, with autostart being on, by
definition if recovery / verification phases can be completed, the
cluster should change its state to RUNNING.

The corresponding test (introduced in 58774fb6 "master: new option to
automatically start a new cluster") was not checking that.

* origin/master:
  neoctl: make 'print ids' command display time of TIDs
  mysql: force _getNextTID() to use appropriate/whole index

Currently, the command "neoctl [arguments] print ids" has the following output:

    last_oid = 0x...
    last_tid = 0x...
    last_ptid = ...

or

    backup_tid = 0x...
    last_tid = 0x...
    last_ptid = ...

depending on whether the cluster is in normal or backup mode.

This is extremely unreadable since the admin is often interested in the time that corresponds to each tid. Now the output is:

    last_oid = 0x...
    last_tid = 0x... (yyyy-mm-dd hh:mm:ss.ssssss)
    last_ptid = ...

or

    backup_tid = 0x... (yyyy-mm-dd hh:mm:ss.ssssss)
    last_tid = 0x... (yyyy-mm-dd hh:mm:ss.ssssss)
    last_ptid = ...

/reviewed-on nexedi/neoppod!2

Similarly to 13911ca3 on the same instance after MariaDB was upgraded to
10.1.17 the following query, even after `OPTIMIZE TABLE obj`, started to execute
very slowly:

    MariaDB [(none)]> SELECT tid FROM neo1.obj WHERE `partition`=5 AND oid=79613 AND tid>268707071353462798 ORDER BY tid LIMIT 1;
    +--------------------+
    | tid                |
    +--------------------+
    | 268707072758797063 |
    +--------------------+
    1 row in set (4.82 sec)

Both explain and analyze says the query will/is using `partition` key but only partially (note key_len is only 10, not 18):

    MariaDB [(none)]> SHOW INDEX FROM neo1.obj;
    +-------+------------+-----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
    | Table | Non_unique | Key_name  | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment | Index_comment |
    +-------+------------+-----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
    | obj   |          0 | PRIMARY   |            1 | partition   | A         |    28755928 |     NULL | NULL   |      | BTREE      |         |               |
    | obj   |          0 | PRIMARY   |            2 | tid         | A         |    28755928 |     NULL | NULL   |      | BTREE      |         |               |
    | obj   |          0 | PRIMARY   |            3 | oid         | A         |    28755928 |     NULL | NULL   |      | BTREE      |         |               |
    | obj   |          0 | partition |            1 | partition   | A         |    28755928 |     NULL | NULL   |      | BTREE      |         |               |
    | obj   |          0 | partition |            2 | oid         | A         |    28755928 |     NULL | NULL   |      | BTREE      |         |               |
    | obj   |          0 | partition |            3 | tid         | A         |    28755928 |     NULL | NULL   |      | BTREE      |         |               |
    | obj   |          1 | data_id   |            1 | data_id     | A         |    28755928 |     NULL | NULL   | YES  | BTREE      |         |               |
    +-------+------------+-----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
    7 rows in set (0.00 sec)

    MariaDB [(none)]> explain SELECT tid FROM neo1.obj WHERE `partition`=5 AND oid=79613 AND tid>268707071353462798 ORDER BY tid LIMIT 1;
    +------+-------------+-------+------+-------------------+-----------+---------+-------------+------+--------------------------+
    | id   | select_type | table | type | possible_keys     | key       | key_len | ref         | rows | Extra                    |
    +------+-------------+-------+------+-------------------+-----------+---------+-------------+------+--------------------------+
    |    1 | SIMPLE      | obj   | ref  | PRIMARY,partition | partition | 10      | const,const |    2 | Using where; Using index |
    +------+-------------+-------+------+-------------------+-----------+---------+-------------+------+--------------------------+
    1 row in set (0.00 sec)

    MariaDB [(none)]> analyze SELECT tid FROM neo1.obj WHERE `partition`=5 AND oid=79613 AND tid>268707071353462798 ORDER BY tid LIMIT 1;
    +------+-------------+-------+------+-------------------+-----------+---------+-------------+------+------------+----------+------------+--------------------------+
    | id   | select_type | table | type | possible_keys     | key       | key_len | ref         | rows | r_rows     | filtered | r_filtered | Extra                    |
    +------+-------------+-------+------+-------------------+-----------+---------+-------------+------+------------+----------+------------+--------------------------+
    |    1 | SIMPLE      | obj   | ref  | PRIMARY,partition | partition | 10      | const,const |    2 | 9741121.00 |   100.00 |       0.00 | Using where; Using index |
    +------+-------------+-------+------+-------------------+-----------+---------+-------------+------+------------+----------+------------+--------------------------+
    1 row in set (4.93 sec)

By explicitly forcing (partition, oid, tid) index usage which is precisely designed to serve this and similar queries can avoid the query from being slow:

    MariaDB [(none)]> analyze SELECT tid FROM neo1.obj FORCE INDEX(`partition`) WHERE `partition`=5 AND oid=79613 AND tid>268707071353462798 ORDER BY tid LIMIT 1;
    +------+-------------+-------+-------+---------------+-----------+---------+------+------+--------+----------+------------+--------------------------+
    | id   | select_type | table | type  | possible_keys | key       | key_len | ref  | rows | r_rows | filtered | r_filtered | Extra                    |
    +------+-------------+-------+-------+---------------+-----------+---------+------+------+--------+----------+------------+--------------------------+
    |    1 | SIMPLE      | obj   | range | partition     | partition | 18      | NULL |    2 |   1.00 |   100.00 |     100.00 | Using where; Using index |
    +------+-------------+-------+-------+---------------+-----------+---------+------+------+--------+----------+------------+--------------------------+
    1 row in set (0.00 sec)

/cc @jm, @vpelltier, @Tyagov

/reviewed-on nexedi/neoppod!1

* origin/master:
  Add support for latest versions of ZODB (4.4.3 & 5.0.1)

Many patches have been merged upstream :)

A notable change is that lastTransaction() does not ping the master anymore
(but it still causes a connection to the master if the client is disconnected).

After partitions were dropped with TokuDB, we had a case where MariaDB 10.1.14
stopped using the most appropriate index.

MariaDB [neo0]> explain SELECT DISTINCT data_id FROM obj WHERE `partition`=5;
+------+-------------+-------+-------+-------------------+---------+---------+------+------+---------------------------------------+
| id   | select_type | table | type  | possible_keys     | key     | key_len | ref  | rows | Extra                                 |
+------+-------------+-------+-------+-------------------+---------+---------+------+------+---------------------------------------+
|    1 | SIMPLE      | obj   | range | PRIMARY,partition | data_id | 11      | NULL |   10 | Using where; Using index for group-by |
+------+-------------+-------+-------+-------------------+---------+---------+------+------+---------------------------------------+
MariaDB [neo0]> SELECT SQL_NO_CACHE DISTINCT data_id FROM obj WHERE `partition`=5;
Empty set (1 min 51.47 sec)

Expected:

MariaDB [neo1]> explain SELECT DISTINCT data_id FROM obj WHERE `partition`=4;
+------+-------------+-------+------+-------------------+---------+---------+-------+------+------------------------------+
| id   | select_type | table | type | possible_keys     | key     | key_len | ref   | rows | Extra                        |
+------+-------------+-------+------+-------------------+---------+---------+-------+------+------------------------------+
|    1 | SIMPLE      | obj   | ref  | PRIMARY,partition | PRIMARY | 2       | const |    1 | Using where; Using temporary |
+------+-------------+-------+------+-------------------+---------+---------+-------+------+------------------------------+
1 row in set (0.00 sec)
MariaDB [neo1]> SELECT SQL_NO_CACHE DISTINCT data_id FROM obj WHERE `partition`=4;
Empty set (0.00 sec)

Restarting the server or 'OPTIMIZE TABLE obj; ' does not help.

Such issue could prevent the cluster to start due to timeouts, by always going
back to RECOVERING state.

Freeing disk space when a cell is dropped will have to be implemented with care,
not only for performance reasons.