Update documentation

BUGS

+Although NEO is considered ready for production use in most cases, there are
+a few bugs to know because they concern basic features of ZODB, or promised
+features of NEO.
+All the listed bugs will be fixed with high priority.
+
+Conflict resolution not fully implemented
+-----------------------------------------
+
+When a cluster has several storage nodes, so-called 'deadlock avoidance' may
+happen to in order to resolve conflicts. In such cases, conflicts will not be
+resolved even if your _p_resolveConflict() method would succeed, leading to a
+normal ConflictError.
+Although this should happen rarely enough not to affect performance, this can
+be an issue if your application can't afford restarting the transaction,
+e.g. because it interacted with external environment.
+
+Master failure not recovered by clients
+---------------------------------------
+
+After a master failure, clients are stuck in a bad state. In particular, caches
+are not flushed, which could lead to data corruption if they reconnect a master.
+This makes secondary masters mostly useless for the moment.
+
+Storage failure or update may lead to POSException or break undoLog()
+---------------------------------------------------------------------
+
+Storage nodes are only queried once at most and if all (for the requested
+partition) failed, the client raises instead of asking the master whether it
+had an up-to-date partition table (and retry if useful).
+In the case of undoLog(), incomplete results may be returned.

README

@@ -14,9 +14,9 @@ A NEO cluster is composed of the following types of nodes:
 
 - "storage" nodes (mandatory, 1 or more)
 
-  Stores data in a MySQL database, preserving history. All available storage
-  nodes are in use simultaneously. This offers redundancy and data distribution.
-  Other storage backends than MySQL are considered for future release.
+  Stores data, preserving history. All available storage nodes are in use
+  simultaneously. This offers redundancy and data distribution.
+  Available backends: MySQL, SQLite
 
 - "admin" nodes (mandatory for startup, optional after)
 
@@ -56,29 +56,43 @@ Requirements
 
 - Python 2.6.x or 2.7.x
 
-- For storage nodes:
+- For storage nodes using MySQL backend:
 
   - MySQLdb: http://sourceforge.net/projects/mysql-python
 
-- For client nodes: ZODB 3.10.x but it should work with ZODB 3.9.x
+- For client nodes: ZODB 3.9.x or later
 
 Installation
 ============
 
-a. Make neo directory available for python to import (for example, by
+a. NEO can be installed like any other egg (see setup.py). Or you can simply
+   make `neo` directory available for Python to import (for example, by
    adding its container directory to the PYTHONPATH environment variable).
 
-b. Choose a cluster name and setup a MySQL database
+b. Write a neo.conf file like the example provided. If you use MySQL,
+   you'll also need create 1 database per storage node.
 
 c. Start all required nodes::
 
-    neomaster --cluster=<cluster name>
-    neostorage --cluster=<cluster name> --database=user:passwd@db
-    neoadmin --cluster=<cluster name>
+    $ neomaster -f neo.conf
+    $ neostorage -f neo.conf -s storage1
+    $ neostorage -f neo.conf -s storage2
+    $ neoadmin -f neo.conf
 
-d. Tell the cluster it can provide service::
+d. Tell the cluster to initialize storage nodes::
 
-    neoctl -a <admin> start
+    $ neoctl -a <admin> start
+
+e. Clients can connect when the cluster is in RUNNING state::
+
+    $ neoctl -a <admin> print cluster
+    RUNNING
+
+f. See `neomigrate` command to import an existing FileStorage database,
+   or `neoctl` command for more administrative tasks.
+
+Alternatively, you can use `neosimple` command to quickly setup a cluster for
+testing.
 
 How to use
 ==========
@@ -134,6 +148,23 @@ This can also be done manually, which helps if your cluster is in bad state:
    don't become in OUT_OF_DATE state.
 - Next stop remaining nodes with a SIGINT or SIGTERM.
 
+Master-slave asynchronous replication
+-------------------------------------
+
+This is the recommanded way to backup a NEO cluster.
+Once a cluster with appropriate `upstream_cluster` & `upstream_masters`
+configuration is started, you can switch it into backup mode
+using::
+
+  neoctl -a <admin> set cluster STARTING_BACKUP
+
+It remembers it is in such mode when it is stopped, and it can be put back into
+normal mode (RUNNING)  by setting it into STOPPING_BACKUP state.
+
+Packs are currently not replicated, which means packing should always be done
+up to a TID that is already fully replicated, so that the backup cluster has a
+full history (and not random holes).
+
 Deployment
 ==========
 
@@ -145,15 +176,15 @@ NEO has no built-in deployment features such as process daemonization. We use
 
   [program:storage_01]
   priority=10
-  command=neostorage -c neo -s storage_01 -f /neo/neo.conf
+  command=neostorage -s storage_01 -f /neo/neo.conf
 
   [program:master_01]
   priority=20
-  command=neomaster -c neo -s master_01 -f /neo/neo.conf
+  command=neomaster -s master_01 -f /neo/neo.conf
 
   [program:admin]
   priority=20
-  command=neoadmin -c neo -s admin -f /neo/neo.conf
+  command=neoadmin -s admin -f /neo/neo.conf
 
 Developers
 ==========

neo.conf

 # Note: Unless otherwise noted, all parameters in this configuration file
 # must be identical for all nodes in a given cluster.
+# This file is optionnal: parameters can be given at the command line.
 
-# Default parameters.
+# Common parameters.
 [DEFAULT]
 # The cluster name
 # This must be set.
-# It must be a name unique to a given cluster, to prevent foreign
+# IMPORTANT: It must be a name unique to a given cluster, to prevent foreign
 # misconfigured nodes from interfering.
-cluster:
+cluster: test
 
 # The list of master nodes
 # Master nodes not not in this list will be rejected by the cluster.
@@ -22,18 +23,25 @@ masters: 127.0.0.1:10000
 #   0 means no redundancy
 #   1 means there is a spare copy of all partitions
 replicas: 1
-# Partitions: How data spreads among storage nodes. This number must be at
-# least equal to the number of storage nodes the cluster contains.
-# IMPORTANT: This must not be changed once the cluster contains data.
-partitions: 20
+# Partitions: How data spreads among storage nodes.
+# IMPORTANT: This can not be changed once the cluster contains data.
+partitions: 12
+# The maximum number of usable storage nodes is: partitions * (replicas + 1)
+
+# Master-slave asynchronous replication
+# The following 2 options are only required if you want to backup another
+# NEO cluster.
+;upstream_cluster: test2
+;upstream_masters: 127.0.0.1:30000
 
 # Individual nodes parameters
 # Some parameters makes no sense to be defined in [DEFAULT] section.
 # They are:
 # bind: The ip:port the node will listen on.
-# database: Storage nodes only.
-#   Syntax for MySQL database is [user[:password]@]database[unix_socket]
-#   Those database must be created manualy.
+# database: Storage nodes only. Syntax for:
+#   - MySQL: [user[:password]@]database[unix_socket]
+#     Database must be created manually.
+#   - SQLite: path
 
 # Admin node
 [admin]

neo/neoctl/app.py

@@ -195,7 +195,7 @@ class TerminalNeoCTL(object):
 
     def checkReplicas(self, params):
         """
-          Test whether partitions have corrupted meta
+          Test whether partitions have corrupted metadata
 
           Any corrupted cell is put in CORRUPTED state, possibly make the
           cluster non operational.