Published parameters as simple storage for generated passwords and NEO cluster name

For performance reasons, the root partition requests subpartitions during
initialization of sections, whereas such processing should normally be done
during the update/install phase.

The consequence is that partitions may be requested whereas they depend on
sections that fail (usually just temporarily, because of missing returned
parameters in the first runs).

For example, the request of zope partitions depends on the generation of
passwords:

1. password generated (__init__)
2. zope partitions requested (__init__)
3. password saved (install)

As long as a failure happens between 2 and 3, zope parameters are always
updated with a different password.

In the case of NEO, the instanciation of zope partitions currently succeeds even
if the list of master nodes is missing (note that there is a minor bug to fix
here: whenever a NEO storage is not the main one, zope processes may start too
early, and the user may have to restart zopes manually). The 'inituser_done'
file is created but zope processes fail to start if NEO is used as main storage,
and all this happens before the password was saved in the root partition
([neo-0-final] failing to install because 'admins' parameter returned yet).

This was never an issue with ZEO because zopes start successfully at the same
time the 'inituser_done' file is created.

One way to solve this could have been to introduce a dummy dependency between
[neo-0-final] and any other section generating a password. Quite ugly and we
also found non-optimal to use a non-backuped file in the root partition to save
such information, whereas we need anyway to publish them for the user.

Therefore, we introduce a new 'publish-early' recipe for accessing and
publishing desired parameters before any request of partitions. Of course,
these must not be dropped by the usual [publish] section, and to avoid having
to repeating them all manually, we have also added a '-extends' option to the
'publish' recipe.

We use the same technique to autogenerate and configure cluster name for NEO,
which helps us in minimizing the number of params one has to pass for
requesting NEO.

In the 'generate.password' recipe, the 'storage-path' can now be empty, when
there's no need to save the generated password in a file.

setup.py

@@ -167,6 +167,7 @@ setup(name=name,
           'proactive = slapos.recipe.proactive:Recipe',
           'publish = slapos.recipe.publish:Recipe',
           'publish.serialised = slapos.recipe.publish:Serialised',
+          'publish-early = slapos.recipe.publish_early:Recipe',
           'publishsection = slapos.recipe.publish:PublishSection',
           'publishurl = slapos.recipe.publishurl:Recipe',
           'readline = slapos.recipe.readline:Recipe',

slapos/recipe/generatepassword.py

@@ -48,6 +48,9 @@ class Recipe(object):
     - storage-path: plain-text persistent storage for password,
                     that can only be accessed by the user
       (default: ${buildout:parts-directory}/${:_buildout_section_name_})
+
+    If storage-path is empty, the recipe does not save the password, which is
+    fine it is saved by other means, e.g. using the publish-early recipe.
   """
 
   def __init__(self, buildout, name, options):
@@ -57,16 +60,17 @@ class Recipe(object):
     except KeyError:
       self.storage_path = options['storage-path'] = os.path.join(
         buildout['buildout']['parts-directory'], name)
+    passwd = None
+    if self.storage_path:
       try:
         with open(self.storage_path) as f:
           passwd = f.read()
       except IOError as e:
         if e.errno != errno.ENOENT:
           raise
-      passwd = None
+        self.update = self.install
     if not passwd:
       passwd = self.generatePassword(int(options_get('bytes', '8')))
-      self.update = self.install
     self.passwd = passwd
     # Password must not go into .installed file, for 2 reasons:
     # security of course but also to prevent buildout to always reinstall.

slapos/recipe/publish.py

@@ -32,15 +32,19 @@ CONNECTION_PARAMETER_STRING = 'connection-'
 
 class Recipe(GenericSlapRecipe):
   def _install(self):
-    publish_dict = dict()
-    options = self.options.copy()
-    del options['recipe']
-    slave_reference = options.pop('-slave-reference', None)
-    for k, v in options.iteritems():
+    publish_dict = {}
+    done = set()
+    extends = [self.name]
+    while extends:
+        name = extends.pop()
+        done.add(name)
+        for k, v in self.buildout[name].iteritems():
           if k[:1] == '-':
-        continue
+            if k == '-extends':
+              extends += set(v.split()) - done
+          elif k != 'recipe':
             publish_dict[k] = v
-    self._setConnectionDict(publish_dict, slave_reference)
+    self._setConnectionDict(publish_dict, self.options.get('-slave-reference'))
     return []
 
   def _setConnectionDict(self, publish_dict, slave_reference=None):

slapos/recipe/publish_early.py

+##############################################################################
+#
+# Copyright (c) 2010 Vifib SARL and Contributors. All Rights Reserved.
+#
+# WARNING: This program as such is intended to be used by professional
+# programmers who take the whole responsibility of assessing all potential
+# consequences resulting from its eventual inadequacies and bugs
+# End users who are looking for a ready-to-use solution with commercial
+# guarantees and support are strongly adviced to contract a Free Software
+# Service Company
+#
+# This program is Free Software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License
+# as published by the Free Software Foundation; either version 3
+# of the License, or (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA  02111-1307, USA.
+#
+##############################################################################
+
+import slapos.slap
+from slapos.recipe.librecipe import unwrap, wrap
+from slapos.recipe.librecipe import GenericSlapRecipe
+
+class Recipe(GenericSlapRecipe):
+  """
+  Early initialization of published parameters.
+
+  The '-init' option defines parameters that should be published before
+  requesting any partitions, and how they are initialized.
+
+  Example:
+
+    [publish-early]
+    recipe = slapos.cookbook:publish-early
+    -init =
+      foo gen-foo:x
+      bar gen-bar:y
+    bar = z
+
+    [gen-foo]
+    ...
+
+    [publish]
+    recipe = slapos.cookbook:publish.serialised
+    -extends = publish-early
+    ...
+
+  ${publish-early:foo} is initialized with the value of the published
+  parameter 'foo', or ${gen-foo:x} if it hasn't been published yet
+  (and in this case, it is published immediately as a way to save the value).
+  ${publish-early:bar} is forced to 'z' (${gen-bar:y} ignored):
+  a line like 'bar = z' is usually rendered conditionally with Jinja2.
+  """
+  def __init__(self, buildout, name, options):
+    GenericSlapRecipe.__init__(self, buildout, name, options)
+    published_dict = None
+    publish = False
+    for line in options['-init'].splitlines():
+      if line:
+        k, v = line.split()
+        if k not in options:
+          if published_dict is None:
+            self.slap.initializeConnection(self.server_url, self.key_file,
+              self.cert_file)
+            computer_partition = self.slap.registerComputerPartition(
+              self.computer_id, self.computer_partition_id)
+            published_dict = unwrap(
+              computer_partition.getConnectionParameterDict())
+            publish_dict = {}
+          try:
+            publish_dict[k] = published_dict[k]
+          except KeyError:
+            section, key = v.split(":")
+            publish_dict[k] = self.buildout[section][key]
+            publish = True
+    if publish:
+      computer_partition.setConnectionDict(wrap(publish_dict))
+    options.update(publish_dict)
+
+  install = update = lambda self: None

software/neoppod/root-common.cfg.in

@@ -38,7 +38,7 @@ parts =
 {% set section_id_list = [] -%}
 [{{ prefix }}request-common]
 <= request-common-base
-config-cluster = {{ dumps(parameter_dict['cluster']) }}
+config-cluster = {{ parameter_dict['cluster'] }}
 {% set replicas = parameter_dict.get('replicas', 0) -%}
 config-partitions = {{ dumps(parameter_dict.get('partitions', 12)) }}
 config-replicas = {{ dumps(replicas) }}

software/neoppod/software-common.cfg

@@ -74,7 +74,7 @@ context =
 
 [root-common]
 <= download-base-neo
-md5sum = f3182000925308bc694d1dab46d8142b
+md5sum = 26193dbb132d340c8ba919a616449a17
 
 [instance-neo-admin]
 <= download-base-neo

stack/erp5/buildout.cfg

@@ -278,8 +278,7 @@ context =
 [template-erp5]
 <= download-base
 filename = instance-erp5.cfg.in
-md5sum = df944bcd8d95d4ce667de91c54ff455b
-
+md5sum = 70820c1608edfc923d57ec2d92aed012
 
 [template-zeo]
 <= download-base
@@ -289,7 +288,7 @@ md5sum = 9670cf63099e2c520017a23defff51a4
 [template-zope]
 <= download-base
 filename = instance-zope.cfg.in
-md5sum = 38551c47a3ee3f4e40d2176301e99482
+md5sum = 35f05cb0f8c8dc9c52bc02984bde49f5
 link-binary =
   ${aspell:location}/bin/aspell
   ${dmtx-utils:location}/bin/dmtxwrite

stack/erp5/instance-erp5.cfg.in

@@ -45,11 +45,13 @@ config-{{ k }} = {{ dumps(v) }}
 {%     do storage_dict.setdefault(zodb['type'], {}).__setitem__(name, zodb.pop('server')) -%}
 {%   endif -%}
 {% endfor -%}
-{% do assert(len(mountpoints) == len(zodb_dict)) %}
+{% do assert(len(mountpoints) == len(zodb_dict)) -%}
+{% set neo = [] -%}
 {% for server_type, server_dict in storage_dict.iteritems() -%}
 {%   if server_type == 'neo' -%}
 {%     set ((name, server_dict),) = server_dict.items() -%}
-{%     do zodb_dict[name].setdefault('storage-dict', {}).__setitem__('name', server_dict['cluster']) -%}
+{%     do neo.append(server_dict.get('cluster')) -%}
+{%     do server_dict.__setitem__('cluster', '${publish-early:neo-cluster}') -%}
 {{     root_common.request_neo(server_dict, 'zodb-neo', 'neo-') }}
 {%   else -%}
 {{     assert(server_type == 'zeo', server_type) -}}
@@ -58,21 +60,38 @@ config-{{ k }} = {{ dumps(v) }}
 {%   endif -%}
 {% endfor -%}
 
-[inituser-password]
-{% set inituser_password = slapparameter_dict.get('inituser-password') -%}
-{% if inituser_password -%}
-passwd = {{ dumps(inituser_password) }}
-{% else -%}
-recipe = slapos.cookbook:generate.password
-{% endif -%}
+[publish-early]
+recipe = slapos.cookbook:publish-early
+-init =
+  inituser-password gen-password:passwd
+  deadlock-debugger-password gen-deadlock-debugger-password:passwd
+{%- if neo %}
+  neo-cluster gen-neo-cluster:name
+{%-  if neo[0] %}
+neo-cluster = {{ neo[0] }}
+{%-  endif %}
+{%- endif %}
+{%- set inituser_password = slapparameter_dict.get('inituser-password') %}
+{%- if inituser_password %}
+inituser-password = {{ inituser_password }}
+{%- endif %}
+{%- set deadlock_debugger_password = slapparameter_dict.get('deadlock-debugger-password') -%}
+{%- if deadlock_debugger_password %}
+deadlock-debugger-password = {{ deadlock_debugger_password }}
+{%- endif %}
 
-[deadlock-debugger-password]
-{% set deadlock_debugger_password = slapparameter_dict.get('deadlock-debugger-password') -%}
-{% if deadlock_debugger_password -%}
-passwd = {{ dumps(deadlock_debugger_password) }}
-{% else -%}
+[gen-password]
 recipe = slapos.cookbook:generate.password
-{% endif -%}
+storage-path =
+
+[gen-deadlock-debugger-password]
+<= gen-password
+
+[gen-neo-cluster-base]
+<= gen-password
+
+[gen-neo-cluster]
+name = neo-${gen-neo-cluster-base:passwd}
 
 [request-zope-base]
 <= request-common
@@ -82,12 +101,12 @@ return =
 config-bt5 = {{ dumps(slapparameter_dict.get('bt5', 'erp5_full_text_myisam_catalog erp5_configurator_standard erp5_configurator_maxma_demo erp5_configurator_ung erp5_configurator_run_my_doc')) }}
 config-bt5-repository-url = {{ dumps(slapparameter_dict.get('bt5-repository-url', local_bt5_repository)) }}
 config-cloudooo-url = ${request-cloudooo:connection-url}
-config-deadlock-debugger-password = ${deadlock-debugger-password:passwd}
+config-deadlock-debugger-password = ${publish-early:deadlock-debugger-password}
 config-developer-list = {{ dumps(slapparameter_dict.get('developer-list', [inituser_login])) }}
 config-hosts-dict = {{ dumps(slapparameter_dict.get('hosts-dict', {})) }}
 config-hostalias-dict = {{ dumps(slapparameter_dict.get('hostalias-dict', {})) }}
 config-inituser-login = {{ dumps(inituser_login) }}
-config-inituser-password = ${inituser-password:passwd}
+config-inituser-password = ${publish-early:inituser-password}
 config-kumofs-url = ${request-memcached-persistent:connection-url}
 config-memcached-url = ${request-memcached-volatile:connection-url}
 config-mysql-test-url-list = ${request-mariadb:connection-test-database-list}
@@ -98,6 +117,7 @@ config-timezone = {{ dumps(slapparameter_dict.get('timezone', 'UTC')) }}
 config-zodb-dict = {{ dumps(zodb_dict) }}
 {% for server_type, server_dict in storage_dict.iteritems() -%}
 {%   if server_type == 'neo' -%}
+config-neo-cluster = ${publish-early:neo-cluster}
 config-neo-name = {{ server_dict.keys()[0] }}
 config-neo-masters = ${neo-0-final:connection-masters}
 {%   else -%}
@@ -191,12 +211,11 @@ return = site_url
 
 [publish]
 recipe = slapos.cookbook:publish.serialised
+-extends = publish-early
 {% if 'neo' in storage_dict -%}
 neo-masters = ${neo-0-final:connection-masters}
 neo-admins = ${neo-0-final:connection-admins}
 {% endif  -%}
-deadlock-debugger-password = ${deadlock-debugger-password:passwd}
-inituser-password = ${inituser-password:passwd}
 {#
 Pick any published hosts-dict, they are expected to be identical - and there is
 no way to check here.

stack/erp5/instance-zope.cfg.in

@@ -251,6 +251,7 @@ node-id = {{ dumps(node_id_base ~ '-' ~ index) }}
 {%     do log_list.append(log) -%}
 {%     do zodb['storage-dict'].__setitem__('logfile', log) -%}
 {%     if db_name == slapparameter_dict.get('neo-name') -%}
+{%       do zodb['storage-dict'].__setitem__('name', slapparameter_dict['neo-cluster']) -%}
 {%       do zodb['storage-dict'].__setitem__('master_nodes', slapparameter_dict['neo-masters']) -%}
 {%     endif -%}
 {%   endif -%}