README 4.83 KB
Newer Older
Vincent Pelletier's avatar
Vincent Pelletier committed
1 2
NEO is a distributed, redundant and scalable implementation of ZODB API.
NEO stands for Nexedi Enterprise Object.
Yoshinori Okuji's avatar
Yoshinori Okuji committed
3

4 5
Overview
========
6

7
A NEO cluster is composed of the following types of nodes:
Vincent Pelletier's avatar
Vincent Pelletier committed
8

9
- "master" nodes (mandatory, 1 or more)
10

11 12 13
  Takes care of transactionality. Only one master node is really active
  (the active master node is called "primary master") at any given time,
  extra masters are spares (they are called "secondary masters").
14

15
- "storage" nodes (mandatory, 1 or more)
16

Julien Muchembled's avatar
Julien Muchembled committed
17 18 19
  Stores data in a MySQL database, preserving history. All available storage
  nodes are in use simultaneously. This offers redundancy and data distribution.
  Other storage backends than MySQL are considered for future release.
20

21
- "admin" nodes (mandatory for startup, optional after)
Vincent Pelletier's avatar
Vincent Pelletier committed
22

23 24
  Accepts commands from neoctl tool and transmits them to the
  primary master, and monitors cluster state.
Vincent Pelletier's avatar
Vincent Pelletier committed
25

26
- "client" nodes
Vincent Pelletier's avatar
Vincent Pelletier committed
27

28
  Well... Something needing to store/load data in a NEO cluster.
Vincent Pelletier's avatar
Vincent Pelletier committed
29

Julien Muchembled's avatar
Julien Muchembled committed
30 31 32 33 34 35 36 37 38 39 40
ZODB API is fully implemented except:

- pack: only old revisions of objects are removed for the moment
        (full implementation is considered)
- blobs: not implemented (not considered yet)

There is a simple way to convert FileStorage to NEO and back again.

See also http://www.neoppod.org/links for more detailed information about
features related to scalability.

41 42
Disclaimer
==========
Vincent Pelletier's avatar
Vincent Pelletier committed
43

44 45
In addition of the disclaimer contained in the licence this code is
released under, please consider the following.
Vincent Pelletier's avatar
Vincent Pelletier committed
46

47 48 49 50
NEO does not implement any authentication mechanism between its nodes, and
does not encrypt data exchanged between nodes either.
If you want to protect your cluster from malicious nodes, or your data from
being snooped, please consider encrypted tunelling (such as openvpn).
Vincent Pelletier's avatar
Vincent Pelletier committed
51

52 53
Requirements
============
Vincent Pelletier's avatar
Vincent Pelletier committed
54

55 56 57 58 59 60
- Linux 2.6 or later

- Python 2.4 or later

- For python 2.4: ctypes http://python.net/crew/theller/ctypes/
  (packaged with later python versions)
61

62 63 64
  Note that setup.py does not define any dependency to 'ctypes' so you will
  have to install it explicitely.

65 66 67
- For storage nodes:

  - MySQLdb: http://sourceforge.net/projects/mysql-python
68

69
- For client nodes: ZODB 3.10.x but it should work with ZODB >= 3.4
Vincent Pelletier's avatar
Vincent Pelletier committed
70

Aurel's avatar
Aurel committed
71
Installation
72
============
Aurel's avatar
Aurel committed
73

74 75
a. Make neo directory available for python to import (for example, by
   adding its container directory to the PYTHONPATH environment variable).
Vincent Pelletier's avatar
Vincent Pelletier committed
76

77
b. Choose a cluster name and setup a MySQL database
Vincent Pelletier's avatar
Vincent Pelletier committed
78

79
c. Start all required nodes::
Vincent Pelletier's avatar
Vincent Pelletier committed
80

81 82 83
    neomaster --cluster=<cluster name>
    neostorage --cluster=<cluster name> --database=user:passwd@host
    neoadmin --cluster=<cluster name>
Vincent Pelletier's avatar
Vincent Pelletier committed
84

85
d. Tell the cluster it can provide service::
Vincent Pelletier's avatar
Vincent Pelletier committed
86

87
    neoctl start
Vincent Pelletier's avatar
Vincent Pelletier committed
88 89

How to use
90
==========
Vincent Pelletier's avatar
Vincent Pelletier committed
91

92 93
First make sure Python can import 'neo.client' package.

94 95
In zope
-------
Aurel's avatar
Aurel committed
96

97
a. Edit your zope.conf, add a neo import and edit the `zodb_db` section by
98 99
   replacing its filestorage subsection by a NEOStorage one.
   It should look like::
Aurel's avatar
Aurel committed
100

101 102 103 104 105 106 107 108 109
    %import neo.client
    <zodb_db main>
        # Main FileStorage database
        <NEOStorage>
            master_nodes 127.0.0.1:10000
            name <cluster name>
        </NEOStorage>
        mount-point /
    </zodb_db>
Aurel's avatar
Aurel committed
110

111
b. Start zope
Aurel's avatar
Aurel committed
112

113 114
In a Python script
------------------
Aurel's avatar
Aurel committed
115

116 117 118 119 120
Just create the storage object and play with it::

  from neo.client.Storage import Storage
  s = Storage(master_nodes="127.0.0.1:10010", name="main")
  ...
Aurel's avatar
Aurel committed
121

122 123
"name" and "master_nodes" parameters have the same meaning as in
configuration file.
Aurel's avatar
Aurel committed
124

125 126
Shutting down
-------------
Aurel's avatar
Aurel committed
127

Julien Muchembled's avatar
Julien Muchembled committed
128
There is no administration command yet to stop properly a running cluster.
129
So following manual actions should be done:
Vincent Pelletier's avatar
Vincent Pelletier committed
130

131 132 133 134 135
a. Make sure all clients like Zope instances are stopped, so that cluster
   become idle.
b. Stop all master nodes first with a SIGINT or SIGTERM, so that storage nodes
   don't become in OUT_OF_DATE state.
c. At last stop remaining nodes with a SIGINT or SIGTERM.
Grégory Wisniewski's avatar
Grégory Wisniewski committed
136

137 138
Deployment
==========
Grégory Wisniewski's avatar
Grégory Wisniewski committed
139 140

NEO has no built-in deployment features such as process daemonization. We use
Julien Muchembled's avatar
Julien Muchembled committed
141
`supervisor <http://supervisord.org/>`_ with configuration like below::
Grégory Wisniewski's avatar
Grégory Wisniewski committed
142

143 144
  [group:neo]
  programs=master_01,storage_01,admin
Grégory Wisniewski's avatar
Grégory Wisniewski committed
145

146 147 148 149
  [program:master_01]
  priority=1
  command=neomaster -c neo -s master_01 -f /neo/neo.conf
  user=neo
Grégory Wisniewski's avatar
Grégory Wisniewski committed
150

151 152 153 154
  [program:storage_01]
  priority=2
  command=neostorage -c neo -s storage_01 -f /neo/neo.conf
  user=neo
Grégory Wisniewski's avatar
Grégory Wisniewski committed
155

156 157 158 159
  [program:admin]
  priority=3
  command=neoadmin -c neo -s admin -f /neo/neo.conf
  user=neo
Julien Muchembled's avatar
Julien Muchembled committed
160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178

Developers
==========

Developers interested in NEO may refer to
`NEO Web site <http://www.neoppod.org/>`_ and subscribe to following mailing
lists:

- `neo-users <http://mail.tiolive.com/mailman/listinfo/neo-users>`_:
  users discussion
- `neo-dev <http://mail.tiolive.com/mailman/listinfo/neo-dev>`_:
  developers discussion
- `neo-report <http://mail.tiolive.com/mailman/listinfo/neo-report>`_:
  automated test results (read-only list)

Commercial Support
==================

Nexedi provides commercial support for NEO: http://www.nexedi.com/