Merge branch 'cliff'

documentation/slapos.usage.rst

-=========================
-SlapOS command line usage
-=========================
-
-
-Note:
------
- * Default configuration file of "Node" commands (slapos node, slapos supervisor) if not explicitly defined as first argument is:
-    /etc/opt/slapos/slapos.cfg or value of SLAPOS_CONFIGURATION environment variable if exists.
-
- * Default configuration file of "Client" commands (slapos request, slapos supply, ...) if not explicitly defined as first argument is:
-    ~/.slapos/slapos.cfg or value of SLAPOS_CLIENT_CONFIGURATION environment variable if exists.
-
- * Default log file for Node commands is /var/log/[slapos-node-software.log | slapos-node-instance.log | slapos-node-report.log]. This one requires working log in slapgrid, currently log/console is a total mess.
-
- * Default pid file for Node commands is: /var/run/[slapos-node-software.pid | slapos-node-instance.pid | slapos-node-report.pid].
-
- * Default SlapOS Master is http://www.vifib.net. It can be changed by altering configuration files.
-
-
-
-General commands
-----------------
-
-slapos
-~~~~~~
-Display help/usage.
-
-
-
-SlapOS Client commands
-----------------------
-
-Those commands are used by clients (as human beings or programs) to manage their own instances.
-
-slapos request
-~~~~~~~~~~~~~~
-Usage:
-  slapos request [slapos_configuration] <reference> <software_alias | software-url> [--node id=<computer guid> region=<region> network-type=<network> | location/to/node.json] [--configuration foo=value1 bar=value2 | location/to/configuration.json ] [--type type] [--slave]
-
-Request an instance and get status and parameters of instance.
-
-Examples:
- * Request a wordpress instance named "mybeautifulinstance" on Node named "COMP-12345":
-     slapos request mybeautifulinstance wordpress --node id=COMP-12345
-
- * Request a kvm instance named "mykvm" on Node named "COMP-12345", specifying nbd-host and nbd-ip parameters:
-     slapos request mykvm kvm --node id=COMP-12345 --configuration nbd-host=debian.nbd.vifib.org nbd-port=1024
-
-XXX Change in slaplib: allow to fetch instance params without changing anything. i.e we should do "slapos request myalreadyrequestedinstance" to fetch connection parameters without erasing previously defined instance parameters.
-
-
-slapos search
-~~~~~~~~~~~~~
-Note: Not yet implemented.
-Usage:
-  slapos search <search parameters ex. computer region, instance reference, source_section, etc.>
-
-Returns visible instances matching search parameters.
-
-
-slapos supply
-~~~~~~~~~~~~~
-Usage:
-   slapos supply <software | software_group> <computer_guid | commputer_group>
-
-Ask installation of a software on a specific node or group of nodes. Nodes will then be ready to accept instances of specified software.
-
-Examples:
- * Ask installation of wordpress Software Release on COMP-12345:
-    slapos supply wordpress COMP-12345
-
-slapos remove
-~~~~~~~~~~~~~
-Usage:
-   slapos remove <software | software_group> <computer_guid | commputer_group>
-
-Ask Removal of a software on a specific node or group of nodes. Existing instances won't work anymore.
-XXX "slapos autounsupply a.k.a slapos cleanup"
-
-Examples:
- * Ask installation of wordpress Software Release on COMP-12345:
-    slapos supply wordpress COMP-12345
-
-slapos autosupply
-~~~~~~~~~~~~~~~~~
-Note: Not yet implemented.
-Usage:
-  slapos autosupply <software | software_group> <computer_guid | computer_group>
-
-Like "slapos suppply", but on-demand. Software will be (re)installed only when at least one instance of this software is requested. When no instance of this software is deployed on the node, it will be uninstalled.
-
-
-slapos console
-~~~~~~~~~~~~~~
-Enter in a python console with slap library imported. See "Slapconsole" section to have detailed documentation.
-
-
-slapos <stop|start|destroy>
-~~~~~~~~~~~~~~~~~~~~~~~~~~~
-Note: Not yet implemented.
-Usage:
-  slapos <stop|start|destroy> <instance reference>
-
-Ask start/stop/destruction of selected instance.
-
-Example:
-  * Ask to stop "mywordpressinstance":
-      slapos stop mywordpressinstance
-
-
-
-SlapOS Node commands
---------------------
-
-This kind of commands are used to control the current SlapOS Node. Those commands are only useful for administrators of Nodes.
-# XXX: add an environment variable for configuration file.
-
-slapos node
-~~~~~~~~~~~
-Display status of Node and if not started, launch supervisor daemon.
-
-Temporary note: equivalent of old slapgrid-supervisord + slapgrid-supervisorctl.
-
-
-slapos node register
-~~~~~~~~~~~~~~~~~~~~
-Usage:
-******
-::
-
-  slapos node register <DESIRED NODE NAME> [--login LOGIN [--password PASSWORD]] [--interface-name INTERFACE] [--master-url URL <--master-url-web URL>] [--partition-number NUMBER] [--ipv4-local-network NETWORK] [--ipv6-interface INTERFACE] [--create-tap] [--dry-run]
-
-If login is not provided, asks for user's vifib account then password.
-
-Node will register itself, if not already done, to the SlapOS Master defined in configuration file, and will generate SlapOS configuration file.
-
-XXX-Cedric should be like this: If desired node name is already taken, will raise an error.
-XXX-Cedric: --master-url-web url will disappear in REST API. Currently, "register" uses SlapOS master web URL to register computer, so it needs the web URL (like http://www.vifib.net)
-
-If Node is already registered (slapos.cfg and certificate already present), issues a warning, backups original configuration and creates new one.
-
-XXX-Cedric should check for IPv6 in selected interface
-
-Parameters:
-***********
---login LOGIN                  Your SlapOS Master login. If not provided, asks it interactively.
---password PASSWORD            Your SlapOS Master password. If not provided, asks it interactively. NOTE: giving password as parameter should be avoided for security reasons.
---interface-name INTERFACE     Use interface as primary interface. IP of Partitions will be added to it. Defaults to "eth0".
---master-url URL               URL of SlapOS Master REST API. defaults to "https://slap.vifib.com".
---master-url-web URL           URL of SlapOS Master web access. defaults to "https://www.vifib.com".
---partition-number NUMBER      Number of partitions that will have your SlapOS Node. defaults to "10".
---ipv4-local-network NETWORK   Subnetwork used to assign local IPv4 addresses. It should be a not used network in order to avoid conflicts. defaults to 10.0.0.0/16.
--t, --create-tap                   Will trigger creation of one virtual "tap" interface per Partition and attach it to primary interface. Requires primary interface to be a bridge. defaults to false. Needed to host virtual machines.
--n, --dry-run                      Don't touch to anything in the filesystem. Used to debug.
-
-Notes:
-******
-  * "IPv6 interface" and "create tap" won't be put at all in the SlapOS Node configuration file if not explicitly written.
-
-Examples:
-*********
-  * Register computer named "mycomputer" to vifib::
-
-      slapos node register mycomputer
-
-  * Register computer named "mycomputer" to vifib using br0 as primary interface, tap0 as IPv6 interface and different local ipv4 subnet::
-
-      slapos node register mycomputer --interface-name br0 --ipv6-interface tap0 --ipv4-local-network 11.0.0.0/16
-
-  * Register computer named "mycomputer" to another SlapOS master accessible via https://www.myownslaposmaster.com, and SLAP webservice accessible via https://slap.myownslaposmaster.com (Note that this address should be the "slap" webservice URL, not web URL)::
-
-      slapos node register mycomputer --master-url https://slap.myownslaposmaster.com --master-url-web https://www.myownslaposmaster.com
-
-XXX-Cedric : To be implemented
-  * Register computer named "mycomputer" to vifib, and ask to create tap interface to be able to host KVMs::
-
-      slapos node register mycomputer --create-tap
-
-
-slapos node software
-~~~~~~~~~~~~~~~~~~~~
-Usage:
-******
-::
-
-  slapos node software [--logfile LOGFILE] [--verbose | -v] [--only_sr URL]  [--all] [CONFIGURATION_FILE]
-
-Run software installation/deletion.
-
-Temporary note: equivalent of old slapgrid-sr.
-# XXX: only_sr should be named ??? (process-only ?)
-# XXX: add a "-vv", very verbose, option.
-
-Parameters:
-***********
---logfile LOGFILE              If specified, will log as well output in the file located at FILE.
---only_sr URL                  Only process one specific Software Release that has been supplied on this Computer. If not supplied: do nothing.
---all                          Process all Software Releases, even already installed.
---verbose, -v                  Be more verbose.
-
-Return values:
-**************
-(Among other standard Python return values)
-0        Everything went fine
-1        At least one software hasn't correctly been installed.
-
-
-slapos node instance
-~~~~~~~~~~~~~~~~~~~~
-Usage:
-******
-::
-
-  slapos node instance [--logfile LOGFILE] [--verbose | -v] [--only_cp PARTITION]  [--all] [CONFIGURATION_FILE]
-
-Temporary note: equivalent of old slapgrid-cp.
-
-Run instances deployment.
-
-Parameters:
-***********
---logfile LOGFILE              If specified, will log as well output in the file located at FILE.
---only_cp PARTITION            Only process one specific Computer Partition, if possible.
---all                          Force processing all Computer Partitions.
---verbose, -v                 Be more verbose.
-
-Return values:
-**************
-(Among other standard Python return values)
-0        Everything went fine
-1        At least one instance hasn't correctly been processed.
-2        At least one promise has failed.
-
-
-slapos node report
-~~~~~~~~~~~~~~~~~~
-Usage:
-******
-::
-
-  slapos node report [--logfile LOGFILE] [--verbose | -v] [CONFIGURATION_FILE]
-
-Run instance reports and garbage collection.
-
-Temporary note: equivalent of old slapgrid-ur.
-
-Parameters:
-***********
---logfile LOGFILE              If specified, will log as well output in the file located at FILE.
---verbose, -v                 Be more verbose.
-
-Return values:
-**************
-(Among other standard Python return values)
-0        Everything went fine
-1        At least one instance hasn't correctly been processed.
-
-
-slapos node <start|stop|restart|tail|status>
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-Usage:
-  slapos node <start|stop|restart|tail|status> <instance>:[process]
-
-Start/Stop/Restart/Show stdout/stderr of instance and/or process.
-
-Examples:
-
- * Start all processes of slappart3:
-     slapos node start slappart3:
-
- * Stop only apache in slappart1:
-     slapos node stop slappart1:apache
-
- * Show stdout/stderr of mysqld in slappart2:
-     slapos node tail slappart2:mysqld
-
-slapos node supervisorctl
-~~~~~~~~~~~~~~~~~~~~~~~~~
-Usage:
-  slapos node supervisorctl
-
-Enter into supervisor console.
-
-slapos node supervisord
-~~~~~~~~~~~~~~~~~~~~~~~
-Usage:
-  slapos node supervisord
-
-Launch, if not already launched, supervisor daemon.
-
-slapos node log
-~~~~~~~~~~~~~~~
-Note: Not yet implemented.
-Usage:
-  slapos node log <software|instance|report>
-
-Display log.

documentation/source/conf.py

@@ -22,7 +22,13 @@ sys.path.append(os.path.abspath('../../'))
 
 # Add any Sphinx extension module names here, as strings. They can be extensions
 # coming with Sphinx (named 'sphinx.ext.*') or your custom ones.
-extensions = ['sphinx.ext.autodoc', 'sphinx.ext.intersphinx', 'sphinx.ext.ifconfig', 'repoze.sphinx.autointerface']
+extensions = [
+    'sphinx.ext.autodoc',
+    'sphinx.ext.intersphinx',
+    'sphinx.ext.ifconfig',
+    'repoze.sphinx.autointerface',
+    'sphinxcontrib.programoutput'
+    ]
 
 # Add any paths that contain templates here, relative to this directory.
 # templates_path = ['_templates']
@@ -45,9 +51,9 @@ copyright = u'2011, Vifib'
 # built documents.
 #
 # The short X.Y version.
-version = '0.21-dev'
+from slapos.version import version
 # The full version, including alpha/beta/rc tags.
-release = '0.21-dev'
+release = version
 
 # The language for content autogenerated by Sphinx. Refer to documentation
 # for a list of supported languages.

documentation/source/slapos

+../../slapos/cli/entry.py
\ No newline at end of file

documentation/source/slapos.usage.rst

+=========================
+SlapOS command line usage
+=========================
+
+
+Notes:
+------
+
+* Default SlapOS Master is http://www.slapos.org. It can be changed by altering configuration files or with the ``--master-url``
+  argument for commands that support it.
+
+* Most commands take a configuration file parameter, provided as ``--cfg /path/to/file.cfg``.
+
+  If no such argument is provided:
+  
+  * "node" commands read configuration from :file:`/etc/opt/slapos/slapos.cfg`, or the file referenced by the
+    ``SLAPOS_CONFIGURATION`` environment variable.
+
+  * likewise, "client" commands (request, supply...) use :file:`~/.slapos/slapos.cfg`, or the ``SLAPOS_CLIENT_CONFIGURATION`` variable.
+
+
+
+XXX TODO document 'alias' for software_url, software_group?, computer_group?
+
+
+
+Common options
+--------------
+
+Without arguments, the ``slapos`` program lists all the available commands and common options.
+
+.. program-output:: python slapos
+
+
+The ``-q`` and ``-v`` options control the verbosity of console output (``-v``: DEBUG, default: INFO, ``-q``: WARNING).
+
+Output to a logfile is not affected, and is the same as ``-v``.
+
+
+
+SlapOS Client commands
+----------------------
+
+These commands are used by clients (as human beings or programs) to manage their own instances.
+
+request
+~~~~~~~
+
+.. program-output:: python slapos help request
+
+Examples
+
+* Request a wordpress instance named "mybeautifulinstance" on Node named "COMP-12345"::
+
+    $ slapos request mybeautifulinstance wordpress --node id=COMP-12345
+
+* Request a kvm instance named "mykvm" on Node named "COMP-12345", specifying nbd-host and nbd-ip parameters::
+
+    $ slapos request mykvm kvm --node id=COMP-12345 --configuration \
+        nbd-host=debian.nbd.vifib.net nbd-port=1024
+
+XXX Change in slaplib: allow to fetch instance params without changing anything. i.e we should do "slapos request myalreadyrequestedinstance" to fetch connection parameters without erasing previously defined instance parameters.
+
+
+..
+  search
+  ~~~~~~
+  Note: Not yet implemented.
+  Usage:
+    slapos search <search parameters ex. computer region, instance reference, source_section, etc.>
+  
+  Returns visible instances matching search parameters.
+
+
+supply
+~~~~~~
+
+.. program-output:: python slapos help supply
+
+Ask installation of a software on a specific node or group of nodes. Nodes will then be ready to accept instances of specified software.
+
+Examples
+
+* Ask installation of wordpress Software Release on COMP-12345::
+
+    $ slapos supply wordpress COMP-12345
+
+
+remove
+~~~~~~
+
+.. program-output:: python slapos help remove
+
+Ask Removal of a software from a specific node or group of nodes. Existing instances won't work anymore.
+
+XXX "slapos autounsupply a.k.a slapos cleanup"
+
+Examples
+
+* Ask installation of wordpress Software Release on COMP-12345::
+
+    $ slapos supply wordpress COMP-12345
+
+
+..
+  autosupply
+  ~~~~~~~~~~
+  Note: Not yet implemented.
+  Usage:
+    slapos autosupply <software | software_group> <computer_guid | computer_group>
+  
+  Like "slapos suppply", but on-demand. Software will be (re)installed only when at least one instance of this software is requested. When no instance of this software is deployed on the node, it will be uninstalled.
+
+
+console
+~~~~~~~
+
+.. program-output:: python slapos help console
+
+
+
+..
+  <stop|start|destroy>
+  ~~~~~~~~~~~~~~~~~~~~
+  Note: Not yet implemented.
+  Usage:
+    slapos <stop|start|destroy> <instance reference>
+  
+  Ask start/stop/destruction of selected instance.
+  
+  Example:
+  
+    * Ask to stop "mywordpressinstance"::
+  
+        $ slapos stop mywordpressinstance
+
+
+
+SlapOS Node commands
+--------------------
+
+This group of commands is used to control the current SlapOS Node. They are only useful to Node administrators.
+
+node, node status
+~~~~~~~~~~~~~~~~~
+
+These are aliases for ``node supervisorctl status``.
+It displays the status of the node, also running the supervisor daemon if needed.
+
+.. program-output:: python slapos help node supervisorctl status
+
+
+node register
+~~~~~~~~~~~~~
+
+.. program-output:: python slapos help node register
+
+
+If login is not provided, asks for user's SlapOS Master account then password.
+
+Node will register itself, if not already done, to the SlapOS Master defined in configuration file, and will generate SlapOS configuration file.
+
+XXX-Cedric should be like this: If desired node name is already taken, will raise an error.
+XXX-Cedric: --master-url-web url will disappear in REST API. Currently, "register" uses SlapOS master web URL to register computer, so it needs the web URL (like http://www.slapos.org)
+
+If Node is already registered (slapos.cfg and certificate already present), issues a warning, backups original configuration and creates new one.
+
+XXX-Cedric should check for IPv6 in selected interface
+
+
+Parameters:
+***********
+--login LOGIN                  Your SlapOS Master login. If not provided, asks it interactively.
+--password PASSWORD            Your SlapOS Master password. If not provided, asks it interactively. NOTE: giving password as parameter should be avoided for security reasons.
+--interface-name INTERFACE     Use interface as primary interface. IP of Partitions will be added to it. Defaults to "eth0".
+--master-url URL               URL of SlapOS Master REST API. defaults to "https://slap.vifib.com".
+--master-url-web URL           URL of SlapOS Master web access. defaults to "https://www.vifib.com".
+--partition-number NUMBER      Number of partitions that will have your SlapOS Node. defaults to "10".
+--ipv4-local-network NETWORK   Subnetwork used to assign local IPv4 addresses. It should be a not used network in order to avoid conflicts. defaults to 10.0.0.0/16.
+-t, --create-tap                   Will trigger creation of one virtual "tap" interface per Partition and attach it to primary interface. Requires primary interface to be a bridge. defaults to false. Needed to host virtual machines.
+
+
+Notes:
+******
+
+* "IPv6 interface" and "create tap" won't be put at all in the SlapOS Node configuration file if not explicitly written.
+
+Examples
+
+* Register computer named "mycomputer" to SlapOS Master::
+
+    $ slapos node register mycomputer
+
+* Register computer named "mycomputer" to SlapOS Master using br0 as primary interface, tap0 as IPv6 interface and different local ipv4 subnet::
+
+    $ slapos node register mycomputer --interface-name br0 --ipv6-interface tap0 \
+        --ipv4-local-network 11.0.0.0/16
+
+* Register computer named "mycomputer" to another SlapOS master accessible via https://www.myownslaposmaster.com, and SLAP webservice accessible via https://slap.myownslaposmaster.com (Note that this address should be the "slap" webservice URL, not web URL)::
+
+    $ slapos node register mycomputer --master-url https://slap.myownslaposmaster.com \
+        --master-url-web https://www.myownslaposmaster.com
+
+XXX-Cedric : To be implemented
+
+* Register computer named "mycomputer" to SlapOS Master, and ask to create tap interface to be able to host KVMs::
+
+    $ slapos node register mycomputer --create-tap
+
+
+node software
+~~~~~~~~~~~~~
+
+.. program-output:: python slapos help node software
+
+
+Return values:
+**************
+
+(Among other standard Python return values)
+
+* 0     Everything went fine
+* 1     At least one software was not correctly installed.
+
+
+node instance
+~~~~~~~~~~~~~
+
+.. program-output:: python slapos help node instance
+
+
+Return values:
+**************
+
+(Among other standard Python return values)
+
+* 0    Everything went fine
+* 1    At least one instance was not correctly processed.
+* 2    At least one promise has failed.
+
+
+node report
+~~~~~~~~~~~
+
+.. program-output:: python slapos help node report
+
+Run instance reports and garbage collection.
+
+
+Return values:
+**************
+
+(Among other standard Python return values)
+
+* 0     Everything went fine
+* 1     At least one instance hasn't correctly been processed.
+
+
+node start|stop|restart|tail|status
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+::
+
+ usage: slapos node <start|stop|restart|tail|status> [-h] [--cfg CFG] <instance>:[process]
+
+ Start/Stop/Restart/Show stdout/stderr of instance and/or process.
+
+ optional arguments:
+   -h, --help            show this help message and exit
+   --cfg CFG             SlapOS configuration file - defaults to
+                             $SLAPOS_CONFIGURATION or /etc/opt/slapos/slapos.cfg
+
+Examples
+
+* Start all processes of slappart3::
+
+    $ slapos node start slappart3:
+
+* Stop only apache in slappart1::
+
+    $ slapos node stop slappart1:apache
+
+* Show stdout/stderr of mysqld in slappart2::
+
+    $ slapos node tail slappart2:mysqld
+
+
+
+node supervisorctl
+~~~~~~~~~~~~~~~~~~
+
+.. program-output:: python slapos help node supervisorctl
+
+
+node supervisord
+~~~~~~~~~~~~~~~~
+
+.. program-output:: python slapos help node supervisord
+
+
+
+..
+  node log
+  ~~~~~~~~
+  Note: Not yet implemented.
+  Usage:
+    slapos node log <software|instance|report>
+  
+  Display log.
+
+
+
+
+
+SlapOS Miscellaneous commands
+-----------------------------
+
+cache lookup
+~~~~~~~~~~~~
+
+.. program-output:: python slapos help cache lookup
+
+
+Examples
+
+* See if the wordpress Software Release is available in precompiled format for our distribution::
+
+    $ slapos cache lookup http://git.erp5.org/gitweb/slapos.git/blob_plain/refs/tags/slapos-0.156:/software/kvm/software.cfg
+    Software URL: http://git.erp5.org/gitweb/slapos.git/blob_plain/refs/tags/slapos-0.156:/software/kvm/software.cfg
+    MD5:          4410088e11f370503e9d78db4cfa4ec4
+    -------------
+    Available for: 
+    distribution     |   version    |       id       | compatible?
+    -----------------+--------------+----------------+-------------
+    CentOS           |          6.3 |     Final      | no
+    Fedora           |           17 | Beefy Miracle  | no
+    Ubuntu           |        12.04 |    precise     | yes
+    debian           |        6.0.6 |                | no
+    debian           |          7.0 |                | no
+
+
+
+

setup.py

@@ -44,6 +44,7 @@ setup(name=name,
                                    # XML
           'zope.interface', # slap library implementes interfaces
           'zc.buildout',
+          'cliff',
         ] + additional_install_requires,
       extra_requires={'docs': ('Sphinx', 'repoze.sphinx.autointerface'),},
       tests_require=[
@@ -54,21 +55,43 @@ setup(name=name,
                       # accessing templates
       entry_points={
         'console_scripts': [
-          # One entry point to control them all
-          'slapos = slapos.entry:main',
           'slapos-watchdog = slapos.grid.watchdog:main',
-          'slapproxy = slapos.proxy:main',
-          'slapproxy-query = slapos.proxy.query:main',
+          'slapproxy = slapos.cli_legacy.proxy_start:main',
+          'slapos = slapos.cli.entry:main',
           # Deprecated entry points
-          'slapconsole = slapos.client:slapconsole',
-          'slapformat = slapos.format:main',
-          'slapgrid = slapos.grid.slapgrid:run',
-          'slapgrid-sr = slapos.grid.slapgrid:runSoftwareRelease',
-          'slapgrid-cp = slapos.grid.slapgrid:runComputerPartition',
-          'slapgrid-ur = slapos.grid.slapgrid:runUsageReport',
-          'slapgrid-supervisorctl = slapos.grid.svcbackend:supervisorctl',
-          'slapgrid-supervisord = slapos.grid.svcbackend:supervisord',
-          'bang = slapos.bang:main',
+          'slapconsole = slapos.cli_legacy.console:console',
+          'slapformat = slapos.cli_legacy.format:main',
+          'slapgrid-sr = slapos.cli_legacy.slapgrid:runSoftwareRelease',
+          'slapgrid-cp = slapos.cli_legacy.slapgrid:runComputerPartition',
+          'slapgrid-ur = slapos.cli_legacy.slapgrid:runUsageReport',
+          'slapgrid-supervisorctl = slapos.cli_legacy.svcbackend:supervisorctl',
+          'slapgrid-supervisord = slapos.cli_legacy.svcbackend:supervisord',
+          'bang = slapos.cli_legacy.bang:main',
+        ],
+        'slapos.cli': [
+          # Utilities
+          'cache lookup = slapos.cli.cache:CacheLookupCommand',
+          # SlapOS Node commands
+          'node bang = slapos.cli.bang:BangCommand',
+          'node format = slapos.cli.format:FormatCommand',
+          'node register = slapos.cli.register:RegisterCommand',
+          'node supervisord = slapos.cli.supervisord:SupervisordCommand',
+          'node supervisorctl = slapos.cli.supervisorctl:SupervisorctlCommand',
+          'node status = slapos.cli.supervisorctl:SupervisorctlStatusCommand',
+          'node start = slapos.cli.supervisorctl:SupervisorctlStartCommand',
+          'node stop = slapos.cli.supervisorctl:SupervisorctlStopCommand',
+          'node restart = slapos.cli.supervisorctl:SupervisorctlRestartCommand',
+          'node tail = slapos.cli.supervisorctl:SupervisorctlTailCommand',
+          'node report = slapos.cli.slapgrid:ReportCommand',
+          'node software = slapos.cli.slapgrid:SoftwareCommand',
+          'node instance = slapos.cli.slapgrid:InstanceCommand',
+          # SlapOS client commands
+          'console = slapos.cli.console:ConsoleCommand',
+          'proxy start = slapos.cli.proxy_start:ProxyStartCommand',
+          'proxy show = slapos.cli.proxy_show:ProxyShowCommand',
+          'supply = slapos.cli.supply:SupplyCommand',
+          'remove = slapos.cli.remove:RemoveCommand',
+          'request = slapos.cli.request:RequestCommand',
         ]
       },
       test_suite="slapos.tests",

slapos/bang.py

 # -*- coding: utf-8 -*-
+# vim: set et sts=2:
 ##############################################################################
 #
 # Copyright (c) 2011, 2012 Vifib SARL and Contributors.
@@ -28,31 +29,17 @@
 ##############################################################################
 
 import slapos.slap.slap
-import argparse
-import ConfigParser
 
-def main(*args):
-  parser = argparse.ArgumentParser()
-  parser.add_argument("-m", "--message", default='', help="Message for bang.")
-  parser.add_argument("configuration_file", nargs=1, type=argparse.FileType(),
-      help="SlapOS configuration file.")
-  if len(args) == 0:
-    argument = parser.parse_args()
-  else:
-    argument = parser.parse_args(list(args))
-  configuration_file = argument.configuration_file[0]
-  message = argument.message
-  # Loads config (if config specified)
-  configuration = ConfigParser.SafeConfigParser()
-  configuration.readfp(configuration_file)
-  computer_id = configuration.get('slapos', 'computer_id')
-  master_url = configuration.get('slapos', 'master_url')
-  if configuration.has_option('slapos', 'key_file'):
-    key_file = configuration.get('slapos', 'key_file')
+
+def do_bang(configp, message):
+  computer_id = configp.get('slapos', 'computer_id')
+  master_url = configp.get('slapos', 'master_url')
+  if configp.has_option('slapos', 'key_file'):
+    key_file = configp.get('slapos', 'key_file')
   else:
     key_file = None
-  if configuration.has_option('slapos', 'cert_file'):
-    cert_file = configuration.get('slapos', 'cert_file')
+  if configp.has_option('slapos', 'cert_file'):
+    cert_file = configp.get('slapos', 'cert_file')
   else:
     cert_file = None
   slap = slapos.slap.slap()

slapos/cache.py

 # -*- coding: utf-8 -*-
 
 import ast
-import argparse
-import ConfigParser
 import hashlib
 import json
 import re
@@ -16,29 +14,20 @@ from slapos.grid.distribution import patched_linux_distribution
 def maybe_md5(s):
     return re.match('[0-9a-f]{32}', s)
 
-def cache_lookup():
-    parser = argparse.ArgumentParser()
-    parser.add_argument("configuration_file", help="SlapOS configuration file")
-    parser.add_argument("software_url", help="Your software url or MD5 hash")
-    args = parser.parse_args()
 
-    configuration_parser = ConfigParser.SafeConfigParser()
-    configuration_parser.read(args.configuration_file)
+def do_lookup(configp, software_url):
+    cache_dir = configp.get('networkcache', 'download-binary-dir-url')
 
-    configuration_parser.items('networkcache')
-
-    cache_dir = configuration_parser.get('networkcache', 'download-binary-dir-url')
-
-    if maybe_md5(args.software_url):
-        md5 = args.software_url
+    if maybe_md5(software_url):
+        md5 = software_url
     else:
-        md5 = hashlib.md5(args.software_url).hexdigest()
+        md5 = hashlib.md5(software_url).hexdigest()
 
     try:
         response = urllib2.urlopen('%s/%s' % (cache_dir, md5))
     except urllib2.HTTPError as e:
         if e.code == 404:
-            print 'Object not in cache: %s' % args.software_url
+            print 'Object not in cache: %s' % software_url
         else:
             print 'Error during cache lookup: %s (%s)' % (e.code, e.reason)
         sys.exit(10)
@@ -67,4 +56,3 @@ def cache_lookup():
     for os in ostable:
         compatible = 'yes' if networkcache.os_matches(os, linux_distribution) else 'no'
         print '%-16s | %12s | %s | %s' % (os[0], os[1], os[2].center(14), compatible)
-

slapos/cli/bang.py

+# -*- coding: utf-8 -*-
+
+import logging
+
+from slapos.cli.config import ConfigCommand
+from slapos.bang import do_bang
+
+
+class BangCommand(ConfigCommand):
+    """
+    request update on all partitions
+    """
+
+    log = logging.getLogger('bang')
+
+    def get_parser(self, prog_name):
+        ap = super(BangCommand, self).get_parser(prog_name)
+        ap.add_argument('-m', '--message',
+                        help='Message for bang')
+        return ap
+
+    def take_action(self, args):
+        configp = self.fetch_config(args)
+        do_bang(configp, args.message)

slapos/cli/cache.py

+# -*- coding: utf-8 -*-
+
+import logging
+
+from slapos.cli.config import ConfigCommand
+from slapos.cache import do_lookup
+
+
+class CacheLookupCommand(ConfigCommand):
+    """
+    perform a query to the networkcache
+    You can provide either a complete URL to the software release,
+    or a corresponding MD5 hash value.
+
+    The command will report which OS distribution/version have a binary
+    cache of the software release, and which ones are compatible
+    with the OS you are currently running.
+    """
+
+    log = logging.getLogger('cache-lookup')
+
+    def get_parser(self, prog_name):
+        ap = super(CacheLookupCommand, self).get_parser(prog_name)
+        ap.add_argument('software_url',
+                        help='Your software url or MD5 hash')
+        return ap
+
+    def take_action(self, args):
+        configp = self.fetch_config(args)
+        do_lookup(configp, args.software_url)

slapos/cli/command.py

+# -*- coding: utf-8 -*-
+
+import argparse
+
+import cliff
+
+
+class Command(cliff.command.Command):
+    def get_parser(self, prog_name):
+        parser = argparse.ArgumentParser(
+            description=self.get_description(),
+            prog=prog_name,
+            formatter_class=argparse.RawDescriptionHelpFormatter
+        )
+
+        return parser

slapos/cli/config.py

+# -*- coding: utf-8 -*-
+
+import ConfigParser
+import os
+
+from slapos.cli.command import Command
+
+
+class ConfigError(Exception):
+    pass
+
+
+class ConfigCommand(Command):
+    "Base class for commands that require a configuration file"
+
+    log = None
+
+    default_config_var = 'SLAPOS_CONFIGURATION'
+
+    # use this if default_config_var does not exist
+    default_config_path = '/etc/opt/slapos/slapos.cfg'
+
+    def get_parser(self, prog_name):
+        ap = super(ConfigCommand, self).get_parser(prog_name)
+        ap.add_argument('--cfg', help='SlapOS configuration file - ' +
+                                      'defaults to $%s ' % self.default_config_var +
+                                      'or %s' % self.default_config_path)
+        return ap
+
+    def _get_config(self, cfg_path, required=False):
+        """
+        Returns a configuration object if file exists/readable/valid,
+        None otherwise.
+        Will raise an error instead of returning None if required is True.
+        Even if required is False, may still raise an exception from the
+        configparser if the configuration content is very broken.
+        We don't catch that exception as it will clearly show what is
+        wrong with the file.
+        """
+
+        if not os.path.exists(cfg_path):
+            if required:
+                raise ConfigError('Configuration file does not exist: %s' % cfg_path)
+            else:
+                return None
+
+        configp = ConfigParser.SafeConfigParser()
+        if configp.read(cfg_path) != [cfg_path]:
+            # bad permission, etc.
+            if required:
+                raise ConfigError('Cannot parse configuration file: %s' % cfg_path)
+            else:
+                return None
+
+        return configp
+
+    def fetch_config(self, args):
+        if args.cfg:
+            cfg_path = args.cfg
+        else:
+            cfg_path = os.environ.get(self.default_config_var, self.default_config_path)
+
+        cfg_path = os.path.expanduser(cfg_path)
+
+        self.log.debug('Loading config: %s' % cfg_path)
+
+        return self._get_config(cfg_path, required=True)
+
+
+class ClientConfigCommand(ConfigCommand):
+    default_config_var = 'SLAPOS_CLIENT_CONFIGURATION'
+    default_config_path = '~/.slapos/slapos-client.cfg'

slapos/cli/console.py

+# -*- coding: utf-8 -*-
+
+import logging
+
+from slapos.cli.config import ClientConfigCommand
+from slapos.client import init, do_console, ClientConfig
+
+
+class ConsoleCommand(ClientConfigCommand):
+    """
+    python console with slap library imported
+
+    You can play with the global "slap" object and
+    with the global "request" method.
+
+    examples :
+    >>> # Request instance
+    >>> request(kvm, "myuniquekvm")
+    >>> # Request software installation on owned computer
+    >>> supply(kvm, "mycomputer")
+    >>> # Fetch instance informations on already launched instance
+    >>> request(kvm, "myuniquekvm").getConnectionParameter("url")
+    """
+
+    log = logging.getLogger('console')
+
+    def get_parser(self, prog_name):
+        ap = super(ConsoleCommand, self).get_parser(prog_name)
+
+        ap.add_argument('-u', '--master_url',
+                        help='Url of SlapOS Master to use')
+
+        ap.add_argument('-k', '--key_file',
+                        help='SSL Authorisation key file')
+
+        ap.add_argument('-c', '--cert_file',
+                        help='SSL Authorisation certificate file')
+
+        return ap
+
+    def take_action(self, args):
+        configp = self.fetch_config(args)
+        conf = ClientConfig(args, configp)
+        local = init(conf)
+        do_console(local)

slapos/cli/entry.py

+# -*- coding: utf-8 -*-
+
+import logging
+import sys
+
+import cliff
+import cliff.app
+from cliff.app import LOG
+import cliff.commandmanager
+
+import slapos.version
+
+
+class SlapOSCommandManager(cliff.commandmanager.CommandManager):
+
+    def find_command(self, argv):
+        """Given an argument list, find a command and
+        return the processor and any remaining arguments.
+        """
+        # a little cheating, 'slapos node' is not documented by the help command
+        if argv == ['node']:
+            argv = ['node', 'status']
+
+        search_args = argv[:]
+        name = ''
+        while search_args:
+            if search_args[0].startswith('-'):
+                raise ValueError('Invalid command %r' % search_args[0])
+            next_val = search_args.pop(0)
+            name = '%s %s' % (name, next_val) if name else next_val
+            if name in self.commands:
+                cmd_ep = self.commands[name]
+                cmd_factory = cmd_ep.load()
+                return (cmd_factory, name, search_args)
+        else:
+            print >>sys.stderr, ('The command %r does not exist or is not yet implemented.\n'
+                                 'Please have a look at http://community.slapos.org to read documentation or forum.\n'
+                                 'Please also make sure that SlapOS Node is up to date.' % (argv,))
+            sys.exit(5)
+
+
+class SlapOSApp(cliff.app.App):
+
+    #
+    # self.options.verbose_level:
+    #    -q  -> 0 (WARNING)
+    #        -> 1 (INFO)
+    #    -v  -> 2 (DEBUG)
+    #    -vv -> 3 (...)
+    #    etc.
+    #
+
+    log = logging.getLogger('slapos')
+
+    CONSOLE_MESSAGE_FORMAT = '%(message)s'
+    LOG_FILE_MESSAGE_FORMAT = '[%(asctime)s] %(levelname)-8s %(name)s %(message)s'
+
+    def __init__(self):
+        super(SlapOSApp, self).__init__(
+            description='SlapOS client %s' % slapos.version.version,
+            version=slapos.version.version,
+            command_manager=SlapOSCommandManager('slapos.cli'),
+        )
+
+    def build_option_parser(self, *args, **kw):
+        kw.setdefault('argparse_kwargs', {})
+        kw['argparse_kwargs']['conflict_handler'] = 'resolve'
+        parser = super(SlapOSApp, self).build_option_parser(*args, **kw)
+
+        # add two aliases for --log-file (for compatibility with old commands)
+        parser.add_argument(
+            '--log-file', '--logfile', '--log_file',
+            action='store',
+            default=None,
+            help='Specify a file to log output. Only console by default.',
+        )
+
+        # always show tracebacks on errors
+        parser.set_defaults(debug=True)
+
+        return parser
+
+    def initialize_app(self, argv):
+        if self.options.verbose_level > 2:
+            self.log.debug('initialize_app')
+
+    def prepare_to_run_command(self, cmd):
+        if self.options.verbose_level > 2:
+            self.log.debug('prepare_to_run_command %s', cmd.__class__.__name__)
+
+    def clean_up(self, cmd, result, err):
+        if self.options.verbose_level > 2:
+            self.log.debug('clean_up %s', cmd.__class__.__name__)
+
+        if err:
+            self.log.debug('got an error: %s', err)
+
+    def run(self, argv):
+        # same as cliff.App.run except that it won't re-raise
+        # a logged exception
+        try:
+            self.options, remainder = self.parser.parse_known_args(argv)
+            self.configure_logging()
+            self.interactive_mode = not remainder
+            self.initialize_app(remainder)
+        except Exception as err:
+            if hasattr(self, 'options'):
+                debug = self.options.debug
+            else:
+                debug = True
+            if debug:
+                LOG.exception(err)
+                # XXX change from cliff behaviour: avoid
+                #     displaying the exception twice
+                # raise
+            else:
+                LOG.error(err)
+            return 1
+        result = 1
+        if self.interactive_mode:
+            result = self.interact()
+        else:
+            result = self.run_subcommand(remainder)
+        return result
+
+
+def main(argv=sys.argv[1:]):
+    app = SlapOSApp()
+    if not argv:
+        argv = ['-h']
+    return app.run(argv)
+
+
+if __name__ == '__main__':
+    sys.exit(main(sys.argv[1:]))

slapos/cli/format.py

+# -*- coding: utf-8 -*-
+
+import logging
+import sys
+
+from slapos.cli.config import ConfigCommand
+from slapos.format import do_format, FormatConfig, tracing_monkeypatch, UsageError
+
+
+class FormatCommand(ConfigCommand):
+    """
+    create users, partitions and network configuration
+    """
+
+    log = logging.getLogger('format')
+
+    def get_parser(self, prog_name):
+        ap = super(FormatCommand, self).get_parser(prog_name)
+
+        ap.add_argument('-x', '--computer_xml',
+                        help="Path to file with computer's XML. If does not exists, will be created",
+                        default=None)
+
+        ap.add_argument('--computer_json',
+                        help="Path to a JSON version of the computer's XML (for development only).",
+                        default=None)
+
+        ap.add_argument('-i', '--input_definition_file',
+                        help="Path to file to read definition of computer instead of "
+                        "declaration. Using definition file allows to disable "
+                        "'discovery' of machine services and allows to define computer "
+                        "configuration in fully controlled manner.")
+
+        ap.add_argument('-o', '--output_definition_file',
+                        help="Path to file to write definition of computer from "
+                        "declaration.")
+
+        ap.add_argument('-n', '--dry_run',
+                        help="Don't actually do anything.",
+                        default=False,
+                        action="store_true")
+
+        ap.add_argument('--alter_user',
+                        choices=['True', 'False'],
+                        help="Shall slapformat alter user database [default: True]")
+
+        ap.add_argument('--alter_network',
+                        choices=['True', 'False'],
+                        help="Shall slapformat alter network configuration [default: True]")
+
+        ap.add_argument('--now',
+                        help="Launch slapformat without delay",
+                        default=False,
+                        action="store_true")
+
+        return ap
+
+    def take_action(self, args):
+        configp = self.fetch_config(args)
+
+        conf = FormatConfig(logger=self.log)
+
+        conf.mergeConfig(args, configp)
+
+        if not self.app.options.log_file and conf.log_file:
+            # no log file is provided by argparser,
+            # we set up the one from config
+            file_handler = logging.FileHandler(conf.log_file)
+            formatter = logging.Formatter(self.app.LOG_FILE_MESSAGE_FORMAT)
+            file_handler.setFormatter(formatter)
+            self.log.addHandler(file_handler)
+
+        try:
+            conf.setConfig()
+        except UsageError as err:
+            sys.stderr.write(err.message + '\n')
+            sys.stderr.write("For help use --help\n")
+            sys.exit(1)
+
+        tracing_monkeypatch(conf)
+
+        try:
+            do_format(conf=conf)
+        except:
+            self.log.exception('Uncaught exception:')
+            raise

slapos/proxy/query.py → slapos/cli/proxy_show.py

 # -*- coding: utf-8 -*-
-# vim: set et sts=4:
 
 import collections
-import ConfigParser
-from optparse import OptionParser, Option
-import sys
+import hashlib
+import logging
 
 import lxml.etree
+import prettytable
 import sqlite3
 
+from slapos.cli.config import ConfigCommand
+from slapos.proxy import ProxyConfig
 from slapos.proxy.db_version import DB_VERSION
 
 
-
-class Parser(OptionParser):
-    """
-    Parse all arguments.
-    """
-    def __init__(self, usage=None, version=None):
-      """
-      Initialize all options possibles.
+class ProxyShowCommand(ConfigCommand):
     """
-      OptionParser.__init__(self, usage=usage, version=version,
-                            option_list=[
-          Option("-u", "--database-uri",
-                 type=str,
-                 help="URI for sqlite database"),
-          Option('--show-instances',
-                 help="View instance information",
-                 default=False,
-                 action="store_true"),
-          Option('--show-params',
-                 help="View published parameters",
-                 default=False,
-                 action="store_true"),
-          Option('--show-network',
-                 help="View network information",
-                 default=False,
-                 action="store_true"),
-          Option('--show-all',
-                 help="View all information",
-                 default=False,
-                 action="store_true"),
-      ])
-
-    def check_args(self):
+    display proxy instances and parameters
     """
-        Check arguments
-        """
-        (options, args) = self.parse_args()
-        if len(args) < 1:
-            self.error("Incorrect number of arguments")
 
-        return options, args[0]
+    log = logging.getLogger('proxy')
 
-class Config:
-    def setConfig(self, option_dict, configuration_file_path):
-        """
-        Set options given by parameters.
-        """
-        # Set options parameters
-        for option, value in option_dict.__dict__.items():
-          setattr(self, option, value)
+    def get_parser(self, prog_name):
+        ap = super(ProxyShowCommand, self).get_parser(prog_name)
+
+        ap.add_argument('-u', '--database-uri',
+                        help='URI for sqlite database')
 
-        # Load configuration file
-        configuration_parser = ConfigParser.SafeConfigParser()
-        configuration_parser.read(configuration_file_path)
-        # Merges the arguments and configuration
-        for section in ("slapproxy", "slapos"):
-            configuration_dict = dict(configuration_parser.items(section))
-            for key in configuration_dict:
-                if not getattr(self, key, None):
-                    setattr(self, key, configuration_dict[key])
+        ap.add_argument('--computers',
+                        help='view computer information',
+                        action='store_true')
 
-        if not self.database_uri:
-            raise ValueError('database-uri is required.')
+        ap.add_argument('--software',
+                        help='view software releases',
+                        action='store_true')
 
+        ap.add_argument('--partitions',
+                        help='view partitions',
+                        action='store_true')
+
+        ap.add_argument('--slaves',
+                        help='view slave instances',
+                        action='store_true')
+
+        ap.add_argument('--params',
+                        help='view published parameters',
+                        action='store_true')
+
+        ap.add_argument('--network',
+                        help='view network settings',
+                        action='store_true')
+
+        return ap
+
+    def take_action(self, args):
+        configp = self.fetch_config(args)
+        conf = ProxyConfig(logger=self.log)
+        conf.mergeConfig(args, configp)
+        conf.setConfig()
+        do_show(conf=conf)
 
 
-tbl_computer = 'computer' + DB_VERSION
-tbl_software = 'software' + DB_VERSION
 tbl_partition = 'partition' + DB_VERSION
-tbl_partition_network = 'partition_network' + DB_VERSION
-tbl_slave = 'slave' + DB_VERSION
 
-null_str = u"-"
+
+def coalesce(*seq):
+    el = None
+    for el in seq:
+        if el is not None:
+            return el
+    return el
 
 
 def print_table(qry, tablename, skip=None):
@@ -92,50 +76,33 @@ def print_table(qry, tablename, skip=None):
         skip = set()
 
     columns = [c[0] for c in qry.description if c[0] not in skip]
+
     rows = []
     for row in qry.fetchall():
-        line = {}
-        for col in columns:
-            val = row[col]
-            if val is None:
-                val = null_str
-            line[col] = val.strip()
-        rows.append(line)
-
-    max_width = {col: len(col) for col in columns}
-    for row in rows:
-        for col in columns:
-            val = row[col]
-            max_width[col] = max(max_width[col], len(val) if val else 0)
+        rows.append([coalesce(row[col], '-') for col in columns])
 
-    hdr = [col.center(max_width[col]) for col in columns]
+    pt = prettytable.PrettyTable(columns)
+    # https://code.google.com/p/prettytable/wiki/Tutorial
 
-    print
+    for row in rows:
+        pt.add_row(row)
 
     if rows:
         print 'table %s:' % tablename,
-    else:
-        print 'table %s: empty' % tablename
-        return
-
         if skip:
             print 'skipping %s' % ', '.join(skip)
         else:
             print
+    else:
+        print 'table %s: empty' % tablename
+        return
 
-    print ' | '.join(hdr)
-    print '-+-'.join('-'*len(h) for h in hdr)
-
-    for row in rows:
-        cells = [row[col].ljust(max_width[col]) for col in columns]
-        print ' | '.join(cells)
+    print pt.get_string(border=True, padding_width=0, vrules=prettytable.NONE)
 
 
 def print_params(conn):
     cur = conn.cursor()
 
-    print
-
     qry = cur.execute("SELECT reference, partition_reference, software_type, connection_xml FROM %s" % tbl_partition)
     for row in qry.fetchall():
         if not row['connection_xml']:
@@ -150,18 +117,19 @@ def print_params(conn):
             if text and name in ('ssh-key', 'ssh-public-key'):
                 text = text[:20] + '...' + text[-20:]
             print '    %s = %s' % (name, text)
-        print
 
 
 def print_computer_table(conn):
+    tbl_computer = 'computer' + DB_VERSION
     cur = conn.cursor()
     qry = cur.execute("SELECT * FROM %s" % tbl_computer)
     print_table(qry, tbl_computer)
 
 
 def print_software_table(conn):
+    tbl_software = 'software' + DB_VERSION
     cur = conn.cursor()
-    qry = cur.execute("SELECT * FROM %s" % tbl_software)
+    qry = cur.execute("SELECT *, md5(url) as md5 FROM %s" % tbl_software)
     print_table(qry, tbl_software)
 
 
@@ -170,21 +138,16 @@ def print_partition_table(conn):
     qry = cur.execute("SELECT * FROM %s WHERE slap_state<>'free'" % tbl_partition)
     print_table(qry, tbl_partition, skip=['xml', 'connection_xml', 'slave_instance_list'])
 
+
 def print_slave_table(conn):
+    tbl_slave = 'slave' + DB_VERSION
     cur = conn.cursor()
     qry = cur.execute("SELECT * FROM %s" % tbl_slave)
     print_table(qry, tbl_slave, skip=['connection_xml'])
 
 
-def print_tables(conn):
-    print_computer_table(conn)
-    print_software_table(conn)
-    print_partition_table(conn)
-    print_slave_table(conn)
-
-
 def print_network(conn):
-    print
+    tbl_partition_network = 'partition_network' + DB_VERSION
     cur = conn.cursor()
     addr = collections.defaultdict(list)
     qry = cur.execute("""
@@ -202,45 +165,40 @@ def print_network(conn):
         print '%s: %s' % (partition_reference, ', '.join(addresses))
 
 
-
-
-def run(config):
-    conn = sqlite3.connect(config.database_uri)
+def do_show(conf):
+    conn = sqlite3.connect(conf.database_uri)
     conn.row_factory = sqlite3.Row
 
-    fn = []
-
-    if config.show_all or config.show_instances:
-        fn.append(print_tables)
-    if config.show_all or config.show_params:
-        fn.append(print_params)
-    if config.show_all or config.show_network:
-        fn.append(print_network)
-
-    if fn:
-        for f in fn:
-            f(conn)
-    else:
-        print 'usage: %s [ --show-params | --show-network | --show-instances | --show-all ]' % sys.argv[0]
-
+    conn.create_function('md5', 1, lambda s: hashlib.md5(s).hexdigest())
 
+    print_all = not any(
+            [
+                conf.computers,
+                conf.software,
+                conf.partitions,
+                conf.slaves,
+                conf.params,
+                conf.network,
+                ]
+            )
 
-def main():
-  "Run default configuration."
-  usage = "usage: %s [options] CONFIGURATION_FILE" % sys.argv[0]
-
-  try:
-    # Parse arguments
-    config = Config()
-    config.setConfig(*Parser(usage=usage).check_args())
-
-    run(config)
-    return_code = 0
-  except SystemExit, err:
-    # Catch exception raise by optparse
-    return_code = err
-
-  sys.exit(return_code)
-
+    if print_all or conf.computers:
+        print_computer_table(conn)
+        print
+    if print_all or conf.software:
+        print_software_table(conn)
+        print
+    if print_all or conf.partitions:
+        print_partition_table(conn)
+        print
+    if print_all or conf.slaves:
+        print_slave_table(conn)
+        print
+    if print_all or conf.params:
+        print_params(conn)
+        print
+    if print_all or conf.network:
+        print_network(conn)
+        print
 
 

slapos/cli/proxy_start.py

+# -*- coding: utf-8 -*-
+
+import logging
+
+from slapos.cli.config import ConfigCommand
+from slapos.proxy import do_proxy, ProxyConfig
+
+
+class ProxyStartCommand(ConfigCommand):
+    """
+    minimalist, stand-alone SlapOS Master
+    """
+
+    log = logging.getLogger('proxy')
+
+    def get_parser(self, prog_name):
+        ap = super(ProxyStartCommand, self).get_parser(prog_name)
+
+        ap.add_argument('-u', '--database-uri',
+                        help='URI for sqlite database')
+
+        return ap
+
+    def take_action(self, args):
+        configp = self.fetch_config(args)
+
+        conf = ProxyConfig(logger=self.log)
+
+        conf.mergeConfig(args, configp)
+
+        if not self.app.options.log_file and hasattr(conf, 'log_file'):
+            # no log file is provided by argparser,
+            # we set up the one from config
+            file_handler = logging.FileHandler(conf.log_file)
+            formatter = logging.Formatter(self.app.LOG_FILE_MESSAGE_FORMAT)
+            file_handler.setFormatter(formatter)
+            self.log.addHandler(file_handler)
+
+        conf.setConfig()
+
+        do_proxy(conf=conf)
+

slapos/cli/register.py

+# -*- coding: utf-8 -*-
+
+import logging
+import sys
+
+from slapos.cli.command import Command
+from slapos.register.register import do_register, RegisterConfig
+
+
+class RegisterCommand(Command):
+    """
+    register a node in the SlapOS cloud
+    """
+
+    log = logging.getLogger('register')
+
+    def get_parser(self, prog_name):
+        ap = super(RegisterCommand, self).get_parser(prog_name)
+
+        ap.add_argument('node_name',
+                        help='Name of the node')
+
+        ap.add_argument('--interface-name',
+                        help='Interface name to access internet',
+                        default='eth0')
+
+        ap.add_argument('--master-url',
+                        help='URL of SlapOS master',
+                        default='https://slap.vifib.com')
+
+        ap.add_argument('--master-url-web',
+                        help='URL of SlapOS Master webservice to register certificates',
+                        default='https://www.slapos.org')
+
+        ap.add_argument('--partition-number',
+                        help='Number of partition on computer',
+                        default='10',
+                        type=int)
+
+        ap.add_argument('--ipv4-local-network',
+                        help='Base of ipv4 local network',
+                        default='10.0.0.0/16')
+
+        ap.add_argument('--ipv6-interface',
+                        help='Interface name to get ipv6',
+                        default='')
+
+        ap.add_argument('--login',
+                        help='User login on SlapOS Master webservice')
+
+        ap.add_argument('--password',
+                        help='User password on SlapOs Master webservice')
+
+        ap.add_argument('-t', '--create-tap',
+                        help='Will trigger creation of one virtual "tap" interface per '
+                             'Partition and attach it to primary interface. Requires '
+                             'primary interface to be a bridge. defaults to false. '
+                             'Needed to host virtual machines.',
+                        default=False,
+                        action='store_true')
+
+        ap.add_argument('-n', '--dry-run',
+                        help='Simulate the execution steps',
+                        default=False,
+                        action='store_true')
+
+        return ap
+
+    def take_action(self, args):
+        try:
+            conf = RegisterConfig(logger=self.log)
+            conf.setConfig(args)
+            return_code = do_register(conf)
+        except SystemExit as err:
+            return_code = err
+
+        sys.exit(return_code)

slapos/cli/remove.py

+# -*- coding: utf-8 -*-
+
+import logging
+
+from slapos.cli.config import ClientConfigCommand
+from slapos.client import init, do_remove, ClientConfig
+
+
+class RemoveCommand(ClientConfigCommand):
+    """
+    remove a Software from a node
+    """
+
+    log = logging.getLogger('remove')
+
+    def get_parser(self, prog_name):
+        ap = super(RemoveCommand, self).get_parser(prog_name)
+
+        ap.add_argument('software_url',
+                        help='Your software url')
+
+        ap.add_argument('node',
+                        help="Target node")
+
+        return ap
+
+    def take_action(self, args):
+        configp = self.fetch_config(args)
+        conf = ClientConfig(args, configp)
+        local = init(conf)
+        do_remove(args.software_url, args.node, local)

slapos/cli/request.py

+# -*- coding: utf-8 -*-
+
+import logging
+
+from slapos.cli.config import ClientConfigCommand
+from slapos.client import init, do_request, ClientConfig
+
+
+def parse_option_dict(options):
+    """
+    Parse a list of option strings like foo=bar baz=qux and return a dictionary.
+    Will raise if keys are repeated.
+    """
+    ret = {}
+    for option_pair in (options or []):
+        key, value = option_pair.split('=', 1)
+        if key in ret:
+            raise ValueError("Multiple values provided for the same key '%s'" % key)
+        ret[key] = value
+    return ret
+
+
+class RequestCommand(ClientConfigCommand):
+    """request an instance and get status and parameters of instance"""
+
+    log = logging.getLogger('request')
+
+    def get_parser(self, prog_name):
+        ap = super(RequestCommand, self).get_parser(prog_name)
+
+        ap.add_argument('reference',
+                        help='Your instance reference')
+
+        ap.add_argument('software_url',
+                        help='Your software url')
+        # XXX TODO can this be an MD5 hash?
+        # XXX TODO can we do a minimal check for correctness of this argument?
+        #          the alternative is a silent failure for mistyped/obsolete/invalid URL
+
+        ap.add_argument('--node',
+                        nargs='+',
+                        help="Node request option 'option1=value1 option2=value2'")
+
+        ap.add_argument('--type',
+                        help='Software type to be requested')
+
+        ap.add_argument('--state',
+                        help='State of the requested instance')
+
+        ap.add_argument('--slave',
+                        action='store_true',
+                        help='Ask for a slave instance')
+
+        ap.add_argument('--parameters',
+                        nargs='+',
+                        help="Give your configuration 'option1=value1 option2=value2'")
+
+        return ap
+
+    def take_action(self, args):
+        args.node = parse_option_dict(args.node)
+        args.parameters = parse_option_dict(args.parameters)
+
+        configp = self.fetch_config(args)
+        conf = ClientConfig(args, configp)
+
+        local = init(conf)
+        do_request(conf, local)

slapos/cli/slapgrid.py

+# -*- coding: utf-8 -*-
+
+import logging
+
+from slapos.cli.config import ConfigCommand
+
+from slapos.grid.utils import setRunning, setFinished
+from slapos.grid.slapgrid import (merged_options, check_missing_parameters, check_missing_files,
+                                  random_delay, create_slapgrid_object)
+
+
+class SlapgridCommand(ConfigCommand):
+
+    log = None
+
+    method_name = NotImplemented
+    default_pidfile = NotImplemented
+
+    def get_parser(self, prog_name):
+        ap = super(SlapgridCommand, self).get_parser(prog_name)
+
+        # TODO move more options to the instance, software and report subclasses
+
+        ap.add_argument('--instance-root',
+                        help='The instance root directory location.')
+        ap.add_argument('--software-root',
+                        help='The software_root directory location.')
+        ap.add_argument('--master-url',
+                        help='The master server URL. Mandatory.')
+        ap.add_argument('--computer-id',
+                        help='The computer id defined in the server.')
+        ap.add_argument('--supervisord-socket',
+                        help='The socket supervisor will use.')
+        ap.add_argument('--supervisord-configuration-path',
+                        help='The location where supervisord configuration will be stored.')
+        ap.add_argument('--buildout', default=None,
+                        help='Location of buildout binary.')
+        ap.add_argument('--pidfile',
+                        help='The location where pidfile will be created. '
+                             'Can be provided by configuration file, or defaults '
+                             'to %s' % self.default_pidfile)
+        ap.add_argument('--key_file',
+                        help='SSL Authorisation key file.')
+        ap.add_argument('--cert_file',
+                        help='SSL Authorisation certificate file.')
+        ap.add_argument('--signature_private_key_file',
+                        help='Signature private key file.')
+        ap.add_argument('--master_ca_file',
+                        help='Root certificate of SlapOS master key.')
+        ap.add_argument('--certificate_repository_path',
+                        help='Path to directory where downloaded certificates would be stored.')
+        ap.add_argument('--maximum-periodicity', type=int, default=None,
+                        help='Periodicity at which buildout should be run in instance.')
+        ap.add_argument('--promise-timeout', type=int, default=3,
+                        help='Promise timeout in seconds.')
+        ap.add_argument('--now', action='store_true',
+                        help='Launch slapgrid without delay. Default behavior.')
+        return ap
+
+    def take_action(self, args):
+        configp = self.fetch_config(args)
+        options = merged_options(args, configp)
+
+        check_missing_parameters(options)
+        check_missing_files(options)
+
+        random_delay(options, logger=self.log)
+
+        slapgrid_object = create_slapgrid_object(options, logger=self.log)
+
+        pidfile = options.get('pidfile') or self.default_pidfile
+
+        if pidfile:
+            setRunning(logger=self.log, pidfile=pidfile)
+        try:
+            return getattr(slapgrid_object, self.method_name)()
+        finally:
+            if pidfile:
+                setFinished(pidfile)
+
+
+class SoftwareCommand(SlapgridCommand):
+    """run software installation/deletion"""
+
+    log = logging.getLogger('software')
+
+    method_name = 'processSoftwareReleaseList'
+    default_pidfile = '/opt/slapos/slapgrid-sr.pid'
+
+    def get_parser(self, prog_name):
+        ap = super(SoftwareCommand, self).get_parser(prog_name)
+
+        only = ap.add_mutually_exclusive_group()
+        only.add_argument('--all', action='store_true',
+                          help='Process all Software Releases, even if already installed.')
+        only.add_argument('--only-sr', '--only',
+                          help='Force the update of a single software release (can be full URL or MD5 hash), '
+                               'even if is already installed. This option will make all other '
+                               'sofware releases be ignored.')
+        return ap
+
+
+class InstanceCommand(SlapgridCommand):
+    """run instance deployment"""
+
+    log = logging.getLogger('instance')
+
+    method_name = 'processComputerPartitionList'
+    default_pidfile = '/opt/slapos/slapgrid-cp.pid'
+
+    def get_parser(self, prog_name):
+        ap = super(InstanceCommand, self).get_parser(prog_name)
+
+        only = ap.add_mutually_exclusive_group()
+        only.add_argument('--all', action='store_true',
+                          help='Process all Computer Partitions.')
+        only.add_argument('--only-cp', '--only',
+                          help='Update a single or a list of computer partitions '
+                               '(ie.:slappartX, slappartY), '
+                               'this option will make all other computer partitions be ignored.')
+        return ap
+
+
+class ReportCommand(SlapgridCommand):
+    """run instance reports and garbage collection"""
+
+    log = logging.getLogger('report')
+
+    method_name = 'agregateAndSendUsage'
+    default_pidfile = '/opt/slapos/slapgrid-ur.pid'

slapos/cli/supervisorctl.py

+# -*- coding: utf-8 -*-
+
+import argparse
+import logging
+import os
+
+from slapos.cli.config import ConfigCommand
+from slapos.grid.svcbackend import launchSupervisord
+
+import supervisor.supervisorctl
+
+
+class SupervisorctlCommand(ConfigCommand):
+    """enter into supervisor console, for process management"""
+
+    log = logging.getLogger('supervisorctl')
+
+    def get_parser(self, prog_name):
+        ap = super(SupervisorctlCommand, self).get_parser(prog_name)
+        ap.add_argument('supervisor_args',
+                        nargs=argparse.REMAINDER,
+                        help='parameters passed to supervisorctl')
+        return ap
+
+    def take_action(self, args):
+        configp = self.fetch_config(args)
+        instance_root = configp.get('slapos', 'instance_root')
+        configuration_file = os.path.join(instance_root, 'etc', 'supervisord.conf')
+        launchSupervisord(socket=os.path.join(instance_root, 'supervisord.socket'),
+                          configuration_file=configuration_file,
+                          logger=self.log)
+        supervisor.supervisorctl.main(args=['-c', configuration_file] + args.supervisor_args)
+
+
+class SupervisorctlAliasCommand(SupervisorctlCommand):
+    def take_action(self, args):
+        args.supervisor_args = [self.alias] + args.supervisor_args
+        super(SupervisorctlAliasCommand, self).take_action(args)
+
+
+class SupervisorctlStatusCommand(SupervisorctlAliasCommand):
+    """alias for 'node supervisorctl status'"""
+    alias = 'status'
+
+
+class SupervisorctlStartCommand(SupervisorctlAliasCommand):
+    """alias for 'node supervisorctl start'"""
+    alias = 'start'
+
+
+class SupervisorctlStopCommand(SupervisorctlAliasCommand):
+    """alias for 'node supervisorctl stop'"""
+    alias = 'stop'
+
+
+class SupervisorctlRestartCommand(SupervisorctlAliasCommand):
+    """alias for 'node supervisorctl restart'"""
+    alias = 'restart'
+
+
+class SupervisorctlTailCommand(SupervisorctlAliasCommand):
+    """alias for 'node supervisorctl tail'"""
+    alias = 'tail'

slapos/cli/supervisord.py

+# -*- coding: utf-8 -*-
+
+import logging
+import os
+
+from slapos.cli.config import ConfigCommand
+from slapos.grid.svcbackend import launchSupervisord
+
+
+class SupervisordCommand(ConfigCommand):
+    """launch, if not already running, supervisor daemon"""
+
+    log = logging.getLogger('supervisord')
+
+    def take_action(self, args):
+        configp = self.fetch_config(args)
+        instance_root = configp.get('slapos', 'instance_root')
+        launchSupervisord(socket=os.path.join(instance_root, 'supervisord.socket'),
+                          configuration_file=os.path.join(instance_root, 'etc', 'supervisord.conf'),
+                          logger=self.log)

slapos/cli/supply.py

+# -*- coding: utf-8 -*-
+
+import logging
+
+from slapos.cli.config import ClientConfigCommand
+from slapos.client import init, do_supply, ClientConfig
+
+
+class SupplyCommand(ClientConfigCommand):
+    """
+    supply a Software to a node
+    """
+
+    log = logging.getLogger('supply')
+
+    def get_parser(self, prog_name):
+        ap = super(SupplyCommand, self).get_parser(prog_name)
+
+        ap.add_argument('software_url',
+                        help='Your software url')
+
+        ap.add_argument('node',
+                        help="Target node")
+
+        return ap
+
+    def take_action(self, args):
+        configp = self.fetch_config(args)
+        conf = ClientConfig(args, configp)
+        local = init(conf)
+        do_supply(args.software_url, args.node, local)

slapos/cli_legacy/bang.py

+# -*- coding: utf-8 -*-
+##############################################################################
+#
+# Copyright (c) 2011, 2012 Vifib SARL and Contributors.
+# All Rights Reserved.
+#
+# WARNING: This program as such is intended to be used by professional
+# programmers who take the whole responsibility of assessing all potential
+# consequences resulting from its eventual inadequacies and bugs
+# End users who are looking for a ready-to-use solution with commercial
+# guarantees and support are strongly advised to contract a Free Software
+# Service Company
+#
+# This program is Free Software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License
+# as published by the Free Software Foundation; either version 3
+# of the License, or (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA  02111-1307, USA.
+#
+##############################################################################
+
+import argparse
+import ConfigParser
+
+from slapos.bang import do_bang
+
+
+def main(*args):
+    ap = argparse.ArgumentParser()
+    ap.add_argument('-m', '--message', default='', help='Message for bang.')
+    ap.add_argument('configuration_file', type=argparse.FileType(),
+                    help='SlapOS configuration file.')
+    if args:
+        args = ap.parse_args(list(args))
+    else:
+        args = ap.parse_args()
+    configp = ConfigParser.SafeConfigParser()
+    configp.readfp(args.configuration_file)
+    do_bang(configp, args.message)

slapos/cli_legacy/cache.py

+# -*- coding: utf-8 -*-
+
+import argparse
+import ConfigParser
+
+from slapos.cache import do_lookup
+
+
+def cache_lookup():
+    ap = argparse.ArgumentParser()
+    ap.add_argument("configuration_file", help="SlapOS configuration file")
+    ap.add_argument("software_url", help="Your software url or MD5 hash")
+    args = ap.parse_args()
+
+    configp = ConfigParser.SafeConfigParser()
+    configp.read(args.configuration_file)
+
+    do_lookup(configp, args.software_url)

slapos/cli_legacy/console.py

+# -*- coding: utf-8 -*-
+
+import argparse
+import os
+import textwrap
+
+from slapos.client import ClientConfig, init, do_console
+from slapos.cli_legacy.util import get_config_parser
+
+
+def console():
+  description = textwrap.dedent("""\
+    slapconsole allows you interact with slap API. You can play with the global
+    "slap" object and with the global "request" method.
+
+    examples :
+    >>> # Request instance
+    >>> request(kvm, "myuniquekvm")
+    >>> # Request software installation on owned computer
+    >>> supply(kvm, "mycomputer")
+    >>> # Fetch instance informations on already launched instance
+    >>> request(kvm, "myuniquekvm").getConnectionParameter("url")""")
+
+  ap = argparse.ArgumentParser(description=description,
+                               formatter_class=argparse.RawDescriptionHelpFormatter)
+
+  ap.add_argument('-u', '--master_url',
+                  default=None,
+                  help='Url of SlapOS Master to use.')
+
+  ap.add_argument('-k', '--key_file',
+                  help="SSL Authorisation key file.")
+
+  ap.add_argument('-c', '--cert_file',
+                  help="SSL Authorisation certificate file.")
+
+  ap.add_argument('configuration_file',
+                  help='path to slapos.cfg')
+
+  args = ap.parse_args()
+
+  if not os.path.isfile(args.configuration_file):
+    ap.error("%s: Not found or not a regular file." % args.configuration_file)
+
+  configp = get_config_parser(args.configuration_file)
+  conf = ClientConfig(args, configp)
+  local = init(conf)
+  do_console(local)

slapos/entry.py → slapos/cli_legacy/entry.py

@@ -30,19 +30,20 @@ import argparse
 import ConfigParser
 import os
 import sys
-from slapos.bang import main as bang
-from slapos.client import slapconsole as console
-from slapos.client import request as request
-from slapos.client import remove as remove
-from slapos.client import supply as supply
-from slapos.format import main as format
-from slapos.cache import cache_lookup
-from slapos.grid.slapgrid import runComputerPartition as instance
-from slapos.grid.slapgrid import runSoftwareRelease as software
-from slapos.grid.slapgrid import runUsageReport as report
-from slapos.grid.svcbackend import supervisord
-from slapos.grid.svcbackend import supervisorctl
-from slapos.register.register import main as register
+
+from slapos.cli_legacy.bang import main as bang
+from slapos.cli_legacy.console import console
+from slapos.cli_legacy.request import request
+from slapos.cli_legacy.remove import remove
+from slapos.cli_legacy.supply import supply
+from slapos.cli_legacy.format import main as format
+from slapos.cli_legacy.cache import cache_lookup
+from slapos.cli_legacy.slapgrid import runComputerPartition as instance
+from slapos.cli_legacy.slapgrid import runSoftwareRelease as software
+from slapos.cli_legacy.slapgrid import runUsageReport as report
+from slapos.cli_legacy.svcbackend import supervisord
+from slapos.cli_legacy.svcbackend import supervisorctl
+from slapos.cli_legacy.register import main as register
 from slapos.version import version
 
 # Note: this whole file is a hack. We should better try dedicated library
@@ -60,6 +61,7 @@ class EntryPointNotImplementedError(NotImplementedError):
   def __init__(self, *args, **kw_args):
     NotImplementedError.__init__(self, *args, **kw_args)
 
+
 def checkSlaposCfg():
   """
   Check if a slapos configuration file was given as a argument.
@@ -71,12 +73,13 @@ def checkSlaposCfg():
   for element in sys.argv:
     if '.cfg' in element:
       if os.path.exists(element):
-        configuration = ConfigParser.SafeConfigParser()
-        configuration.read(element)
-        if configuration.has_section('slapos'):
+        configp = ConfigParser.SafeConfigParser()
+        configp.read(element)
+        if configp.has_section('slapos'):
           return True
   return False
 
+
 def checkOption(option):
   """
   Check if a given option is already in call line
@@ -88,11 +91,12 @@ def checkOption(option):
     if key in element:
       return True
   sys.argv.append(key)
-  if len(option) > 1 :
+  if len(option) > 1:
     sys.argv = sys.argv + option[1:]
   return True
 
-def call(fun, config=False, option=None):
+
+def call(fun, config_path=False, option=None):
   """
   Add missing options to sys.argv
   Add config if asked and it is missing
@@ -102,12 +106,13 @@ def call(fun, config=False, option=None):
     option = []
   for element in option:
     checkOption(element)
-  if config:
+  if config_path:
     if not checkSlaposCfg():
-      sys.argv = [sys.argv[0]] + [os.path.expanduser(config)] + sys.argv[1:]
+      sys.argv = [sys.argv[0]] + [os.path.expanduser(config_path)] + sys.argv[1:]
   fun()
   sys.exit(0)
 
+
 def dispatch(command, is_node_command):
   """ Dispatch to correct SlapOS module.
   Here we could use introspection to get rid of the big "if" statements,
@@ -124,34 +129,34 @@ def dispatch(command, is_node_command):
     if command == 'register':
       call(register)
     elif command == 'software':
-      call(software, config=GLOBAL_SLAPOS_CONFIGURATION,
+      call(software, config_path=GLOBAL_SLAPOS_CONFIGURATION,
            option=['--pidfile /opt/slapos/slapgrid-sr.pid'])
     elif command == 'instance':
-      call(instance, config=GLOBAL_SLAPOS_CONFIGURATION,
+      call(instance, config_path=GLOBAL_SLAPOS_CONFIGURATION,
            option=['--pidfile /opt/slapos/slapgrid-cp.pid'])
     elif command == 'report':
-      call(report, config=GLOBAL_SLAPOS_CONFIGURATION,
+      call(report, config_path=GLOBAL_SLAPOS_CONFIGURATION,
            option=['--pidfile /opt/slapos/slapgrid-ur.pid'])
     elif command == 'bang':
-      call(bang, config=GLOBAL_SLAPOS_CONFIGURATION)
+      call(bang, config_path=GLOBAL_SLAPOS_CONFIGURATION)
     elif command == 'format':
-      call(format, config=GLOBAL_SLAPOS_CONFIGURATION, option=['-c', '-v'])
+      call(format, config_path=GLOBAL_SLAPOS_CONFIGURATION, option=['-c', '-v'])
     elif command == 'supervisord':
-      call(supervisord, config=GLOBAL_SLAPOS_CONFIGURATION)
+      call(supervisord, config_path=GLOBAL_SLAPOS_CONFIGURATION)
     elif command == 'supervisorctl':
-      call(supervisorctl, config=GLOBAL_SLAPOS_CONFIGURATION)
+      call(supervisorctl, config_path=GLOBAL_SLAPOS_CONFIGURATION)
     elif command in ['start', 'stop', 'restart', 'status', 'tail']:
       # Again, too hackish
       sys.argv[-2:-2] = [command]
-      call(supervisorctl, config=GLOBAL_SLAPOS_CONFIGURATION)
+      call(supervisorctl, config_path=GLOBAL_SLAPOS_CONFIGURATION)
     else:
       return False
   elif command == 'request':
-    call(request, config=USER_SLAPOS_CONFIGURATION)
+    call(request, config_path=USER_SLAPOS_CONFIGURATION)
   elif command == 'supply':
-    call(supply, config=USER_SLAPOS_CONFIGURATION)
+    call(supply, config_path=USER_SLAPOS_CONFIGURATION)
   elif command == 'remove':
-    call(remove, config=USER_SLAPOS_CONFIGURATION)
+    call(remove, config_path=USER_SLAPOS_CONFIGURATION)
   elif command == 'start':
     raise EntryPointNotImplementedError(command)
   elif command == 'stop':
@@ -159,12 +164,13 @@ def dispatch(command, is_node_command):
   elif command == 'destroy':
     raise EntryPointNotImplementedError(command)
   elif command == 'console':
-    call(console, config=USER_SLAPOS_CONFIGURATION)
+    call(console, config_path=USER_SLAPOS_CONFIGURATION)
   elif command == 'cache-lookup':
-    call(cache_lookup, config=GLOBAL_SLAPOS_CONFIGURATION)
+    call(cache_lookup, config_path=GLOBAL_SLAPOS_CONFIGURATION)
   else:
     return False
 
+
 def main():
   """
   Main entry point of SlapOS Node. Used to dispatch commands to python
@@ -206,19 +212,19 @@ Node subcommands usage:
 
   # Parse arguments
   # XXX remove the "positional arguments" from help message
-  parser = argparse.ArgumentParser(usage=usage)
-  parser.add_argument('command')
-  parser.add_argument('argument_list', nargs=argparse.REMAINDER)
+  ap = argparse.ArgumentParser(usage=usage)
+  ap.add_argument('command')
+  ap.add_argument('argument_list', nargs=argparse.REMAINDER)
 
-  namespace = parser.parse_args()
+  args = ap.parse_args()
   # Set sys.argv for the sub-entry point that we will call
-  command_line = [namespace.command]
-  command_line.extend(namespace.argument_list)
+  command_line = [args.command]
+  command_line.extend(args.argument_list)
   sys.argv = command_line
 
   try:
-    if not dispatch(namespace.command, is_node):
-      parser.print_help()
+    if not dispatch(args.command, is_node):
+      ap.print_help()
       sys.exit(1)
   except EntryPointNotImplementedError, exception:
     print ('The command %s does not exist or is not yet implemented. Please '
@@ -226,4 +232,3 @@ Node subcommands usage:
            'forum. Please also make sure that SlapOS Node is up to '
            'date.' % exception)
     sys.exit(1)
-

slapos/cli_legacy/format.py

+# -*- coding: utf-8 -*-
+##############################################################################
+#
+# Copyright (c) 2010, 2011, 2012 Vifib SARL and Contributors.
+# All Rights Reserved.
+#
+# WARNING: This program as such is intended to be used by professional
+# programmers who take the whole responsibility of assessing all potential
+# consequences resulting from its eventual inadequacies and bugs
+# End users who are looking for a ready-to-use solution with commercial
+# guarantees and support are strongly advised to contract a Free Software
+# Service Company
+#
+# This program is Free Software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License
+# as published by the Free Software Foundation; either version 3
+# of the License, or (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA  02111-1307, USA.
+#
+##############################################################################
+
+import argparse
+import ConfigParser
+import logging
+import sys
+import os
+
+from slapos.format import FormatConfig, UsageError, tracing_monkeypatch, do_format
+
+
+def main(*args):
+  "Run default configuration."
+
+  ap = argparse.ArgumentParser()
+
+  ap.add_argument('-x', '--computer_xml',
+                  help="Path to file with computer's XML. If does not exists, will be created",
+                  default=None)
+
+  ap.add_argument('--computer_json',
+                  help="Path to a JSON version of the computer's XML (for development only).",
+                  default=None)
+
+  ap.add_argument('-l', '--log_file',
+                  help="The path to the log file used by the script.")
+
+  ap.add_argument('-i', '--input_definition_file',
+                  help="Path to file to read definition of computer instead of "
+                  "declaration. Using definition file allows to disable "
+                  "'discovery' of machine services and allows to define computer "
+                  "configuration in fully controlled manner.")
+
+  ap.add_argument('-o', '--output_definition_file',
+                  help="Path to file to write definition of computer from "
+                  "declaration.")
+
+  ap.add_argument('-n', '--dry_run',
+                  help="Don't actually do anything.",
+                  default=False,
+                  action="store_true")
+
+  ap.add_argument('-v', '--verbose',
+                  default=False,
+                  action="store_true",
+                  help="Verbose output.")
+
+  # the console option is actually ignored and not used anymore.
+  ap.add_argument('-c', '--console',
+                  default=False,
+                  action="store_true",
+                  help="Console output.")
+
+  ap.add_argument('--alter_user',
+                  choices=['True', 'False'],
+                  help="Shall slapformat alter user database [default: True]")
+
+  ap.add_argument('--alter_network',
+                  choices=['True', 'False'],
+                  help="Shall slapformat alter network configuration [default: True]")
+
+  ap.add_argument('--now',
+                  help="Launch slapformat without delay",
+                  default=False,
+                  action="store_true")
+
+  ap.add_argument('configuration_file',
+                  help='path to slapos.cfg')
+
+  if args:
+    args = ap.parse_args(list(args))
+  else:
+    args = ap.parse_args()
+
+  logger = logging.getLogger("slapformat")
+  logger.addHandler(logging.StreamHandler())
+
+  if args.verbose:
+    logger.setLevel(logging.DEBUG)
+    logger.debug("Verbose mode enabled.")
+  else:
+    logger.setLevel(logging.INFO)
+
+  conf = FormatConfig(logger=logger)
+
+  configp = ConfigParser.SafeConfigParser()
+  if configp.read(args.configuration_file) != [args.configuration_file]:
+    raise UsageError('Cannot find or parse configuration file: %s' % args.configuration_file)
+
+  conf.mergeConfig(args, configp)
+
+  if conf.log_file:
+    if not os.path.isdir(os.path.dirname(conf.log_file)):
+      # fallback to console only if directory for logs does not exists and
+      # continue to run
+      raise ValueError('Please create directory %r to store %r log file' % (
+        os.path.dirname(conf.log_file), conf.log_file))
+    else:
+      file_handler = logging.FileHandler(conf.log_file)
+      file_handler.setFormatter(logging.Formatter("%(asctime)s - "
+        "%(name)s - %(levelname)s - %(message)s"))
+      conf.logger.addHandler(file_handler)
+      conf.logger.info('Configured logging to file %r' % conf.log_file)
+
+  try:
+    conf.setConfig()
+  except UsageError as exc:
+    sys.stderr.write(exc.message + '\n')
+    sys.stderr.write("For help use --help\n")
+    sys.exit(1)
+
+  tracing_monkeypatch(conf)
+
+  try:
+    do_format(conf=conf)
+  except:
+    conf.logger.exception('Uncaught exception:')
+    raise

slapos/cli_legacy/proxy_start.py

+# -*- coding: utf-8 -*-
+# vim: set et sts=2:
+
+import argparse
+import ConfigParser
+import logging
+import os
+import sys
+
+from slapos.proxy import ProxyConfig, do_proxy
+
+class UsageError(Exception):
+  pass
+
+
+def main():
+  ap = argparse.ArgumentParser()
+
+  ap.add_argument('-l', '--log_file',
+                  help='The path to the log file used by the script.')
+
+  ap.add_argument('-v', '--verbose',
+                  action='store_true',
+                  help='Verbose output.')
+
+  # XXX not used anymore, deprecated
+  ap.add_argument('-c', '--console',
+                  action='store_true',
+                  help='Console output.')
+
+  ap.add_argument('-u', '--database-uri',
+                  help='URI for sqlite database')
+
+  ap.add_argument('configuration_file',
+                  help='path to slapos.cfg')
+
+  args = ap.parse_args()
+
+  logger = logging.getLogger('slapproxy')
+  logger.addHandler(logging.StreamHandler())
+
+  if args.verbose:
+    logger.setLevel(logging.DEBUG)
+  else:
+    logger.setLevel(logging.INFO)
+
+  conf = ProxyConfig(logger=logger)
+
+  configp = ConfigParser.SafeConfigParser()
+  if configp.read(args.configuration_file) != [args.configuration_file]:
+    raise UsageError('Cannot find or parse configuration file: %s' % args.configuration_file)
+
+  conf.mergeConfig(args, configp)
+
+  if conf.log_file:
+    if not os.path.isdir(os.path.dirname(conf.log_file)):
+      raise ValueError('Please create directory %r to store %r log file' % (
+        os.path.dirname(conf.log_file), conf.log_file))
+    file_handler = logging.FileHandler(conf.log_file)
+    file_handler.setFormatter(logging.Formatter("%(asctime)s - %(name)s - %(levelname)s - %(message)s"))
+    logger.addHandler(file_handler)
+    logger.info('Configured logging to file %r' % conf.log_file)
+
+  conf.setConfig()
+
+  try:
+    do_proxy(conf=conf)
+    return_code = 0
+  except SystemExit as err:
+    return_code = err
+
+  sys.exit(return_code)

slapos/cli_legacy/register.py

+# -*- coding: utf-8 -*-
+# vim: set et sts=2:
+##############################################################################
+#
+# Copyright (c) 2012 Vifib SARL and Contributors. All Rights Reserved.
+#
+# WARNING: This program as such is intended to be used by professional
+# programmers who take the whole responsibility of assessing all potential
+# consequences resulting from its eventual inadequacies and bugs
+# End users who are looking for a ready-to-use solution with commercial
+# guarantees and support are strongly advised to contract a Free Software
+# Service Company
+#
+# This program is Free Software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License
+# as published by the Free Software Foundation; either version 3
+# of the License, or (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA  02111-1307, USA.
+#
+##############################################################################
+
+import argparse
+import logging
+import sys
+
+from slapos.register.register import do_register, RegisterConfig
+
+
+def main():
+  ap = argparse.ArgumentParser()
+
+  ap.add_argument('node_name',
+                  help='Name of the node')
+
+  ap.add_argument('--interface-name',
+                  help='Interface name to access internet',
+                  default='eth0')
+
+  ap.add_argument('--master-url',
+                  help='URL of SlapOS master',
+                  default='https://slap.vifib.com')
+
+  ap.add_argument('--master-url-web',
+                  help='URL of SlapOS Master webservice to register certificates',
+                  default='https://www.slapos.org')
+
+  ap.add_argument('--partition-number',
+                  help='Number of partition on computer',
+                  default='10',
+                  type=int)
+
+  ap.add_argument('--ipv4-local-network',
+                  help='Base of ipv4 local network',
+                  default='10.0.0.0/16')
+
+  ap.add_argument('--ipv6-interface',
+                  help='Interface name to get ipv6',
+                  default='')
+
+  ap.add_argument('--login',
+                  help='User login on SlapOS Master webservice')
+
+  ap.add_argument('--password',
+                  help='User password on SlapOs Master webservice')
+
+  ap.add_argument('-t', '--create-tap',
+                  help='Will trigger creation of one virtual "tap" interface per '
+                       'Partition and attach it to primary interface. Requires '
+                       'primary interface to be a bridge. defaults to false. '
+                       'Needed to host virtual machines.',
+                  default=False,
+                  action='store_true')
+
+  ap.add_argument('-n', '--dry-run',
+                  help='Simulate the execution steps',
+                  default=False,
+                  action='store_true')
+
+  args = ap.parse_args()
+
+  if args.password and not args.login:
+    ap.error('Please enter your login with your password')
+
+  logger = logging.getLogger('Register')
+  handler = logging.StreamHandler()
+  logger.setLevel(logging.DEBUG)
+  handler.setLevel(logging.INFO)
+  handler.setFormatter(logging.Formatter('%(levelname)s - %(message)s'))
+  logger.addHandler(handler)
+
+  try:
+    conf = RegisterConfig(logger=logger)
+    conf.setConfig(args)
+    return_code = do_register(conf)
+  except SystemExit as exc:
+    return_code = exc
+
+  sys.exit(return_code)

slapos/cli_legacy/remove.py

+# -*- coding: utf-8 -*-
+
+import argparse
+
+from slapos.client import ClientConfig, init, do_remove
+from slapos.cli_legacy.util import get_config_parser
+
+
+def remove():
+  ap = argparse.ArgumentParser()
+  ap.add_argument('configuration_file',
+                  help='SlapOS configuration file')
+  ap.add_argument('software_url',
+                  help='Your software url')
+  ap.add_argument('node',
+                  help='Target node')
+  args = ap.parse_args()
+
+  configp = get_config_parser(args.configuration_file)
+  conf = ClientConfig(args, configp)
+  local = init(conf)
+  do_remove(args.software_url, args.node, local)

slapos/cli_legacy/request.py

+# -*- coding: utf-8 -*-
+
+import argparse
+
+from slapos.client import ClientConfig, init, do_request
+from slapos.cli_legacy.util import get_config_parser
+
+
+def argToDict(element):
+  """
+  convert a table of string 'key=value' to dict
+  """
+  if element is not None:
+    element_dict = dict([arg.split('=') for arg in element])
+  return element_dict
+
+
+def request():
+  """Run when invoking slapos request. Request an instance."""
+  # Parse arguments and inititate needed parameters
+  # XXX-Cedric: move argument parsing to main entry point
+
+  ap = argparse.ArgumentParser()
+  ap.add_argument('configuration_file',
+                  help='SlapOS configuration file.')
+  ap.add_argument('reference',
+                  help='Your instance reference')
+  ap.add_argument('software_url',
+                  help='Your software url')
+  ap.add_argument('--node',
+                  nargs='*',
+                  help='Node request option '
+                  "'option1=value1 option2=value2'")
+  ap.add_argument('--type',
+                  type=str,
+                  help='Define software type to be requested')
+  ap.add_argument('--slave',
+                  action='store_true', default=False,
+                  help='Ask for a slave instance')
+  ap.add_argument('--configuration',
+                  nargs='*',
+                  help='Give your configuration '
+                  "'option1=value1 option2=value2'")
+
+  args = ap.parse_args()
+
+  if args.configuration:
+    args.parameters = argToDict(args.configuration)
+
+  if args.node:
+    args.node = argToDict(args.node)
+
+  configp = get_config_parser(args.configuration_file)
+  conf = ClientConfig(args, configp)
+  local = init(conf)
+  do_request(conf, local)

slapos/cli_legacy/slapgrid.py

+# -*- coding: utf-8 -*-
+# vim: set et sts=2:
+
+import argparse
+import ConfigParser
+import logging
+import sys
+
+from slapos.grid.utils import setRunning, setFinished
+from slapos.grid.slapgrid import (merged_options, check_missing_parameters,
+                                  check_missing_files, random_delay, create_slapgrid_object)
+
+
+def parse_arguments(*argument_tuple):
+  """Parse arguments and return options dictionary merged with the config file."""
+
+  ap = argparse.ArgumentParser()
+  ap.add_argument('--instance-root',
+                  help='The instance root directory location.')
+  ap.add_argument('--software-root',
+                  help='The software_root directory location.')
+  ap.add_argument('--master-url',
+                  help='The master server URL. Mandatory.')
+  ap.add_argument('--computer-id',
+                  help='The computer id defined in the server.')
+  ap.add_argument('--supervisord-socket',
+                  help='The socket supervisor will use.')
+  ap.add_argument('--supervisord-configuration-path',
+                  help='The location where supervisord configuration will be stored.')
+  ap.add_argument('--buildout', default=None,
+                  help='Location of buildout binary.')
+  ap.add_argument('--pidfile',
+                  help='The location where pidfile will be created.')
+  ap.add_argument('--logfile',
+                  help='The location where slapgrid logfile will be created.')
+  ap.add_argument('--key_file',
+                  help='SSL Authorisation key file.')
+  ap.add_argument('--cert_file',
+                  help='SSL Authorisation certificate file.')
+  ap.add_argument('--signature_private_key_file',
+                  help='Signature private key file.')
+  ap.add_argument('--master_ca_file',
+                  help='Root certificate of SlapOS master key.')
+  ap.add_argument('--certificate_repository_path',
+                  help='Path to directory where downloaded certificates would be stored.')
+  ap.add_argument('-v', '--verbose', action='store_true',
+                  help='Be verbose.')
+  ap.add_argument('--maximum-periodicity', type=int, default=None,
+                  help='Periodicity at which buildout should be run in instance.')
+  ap.add_argument('--promise-timeout', type=int, default=3,
+                  help='Promise timeout in seconds.')
+  ap.add_argument('--now', action='store_true',
+                  help='Launch slapgrid without delay. Default behavior.')
+  ap.add_argument('--all', action='store_true',
+                  help='Launch slapgrid to process all Software Releases '
+                       'and/or Computer Partitions.')
+  ap.add_argument('--only-sr',
+                  help='Force the update of a single software release (use url hash), '
+                       'even if is already installed. This option will make all other '
+                       'sofware releases be ignored.')
+  ap.add_argument('--only-cp',
+                  help='Update a single or a list of computer partitions '
+                       '(ie.:slappartX, slappartY), '
+                       'this option will make all other computer partitions be ignored.')
+
+  ap.add_argument('configuration_file', type=argparse.FileType(),
+                  help='SlapOS configuration file.')
+
+  # Deprecated options
+  ap.add_argument('-c', '--console', action='store_true',
+                  help="Deprecated, doesn't do anything.")
+  ap.add_argument('--develop', action='store_true',
+                  help='Deprecated, same as --all.')
+  ap.add_argument('--only_sr',
+                  help='Deprecated, same as --only-sr.')
+  ap.add_argument('--only_cp',
+                  help='Deprecated, same as --only-cp.')
+  ap.add_argument('--maximal_delay',
+                  help='Deprecated. Will only work from configuration file in the future.')
+
+  if not argument_tuple:
+    args = ap.parse_args()
+  else:
+    args = ap.parse_args(list(argument_tuple))
+
+  return args
+
+
+def setup_logger(options):
+  logger = logging.getLogger(__name__)
+
+  if options.get('logfile'):
+    handler = logging.FileHandler(options['logfile'])
+  else:
+    handler = logging.StreamHandler()
+
+  if options['verbose']:
+    handler.setLevel(logging.DEBUG)
+  else:
+    handler.setLevel(logging.INFO)
+
+  formatter = logging.Formatter(fmt='%(asctime)s %(name)-18s: '
+                                '%(levelname)-8s %(message)s',
+                                datefmt='%Y-%m-%dT%H:%M:%S')
+
+  handler.setFormatter(formatter)
+  logger.addHandler(handler)
+
+  return logger
+
+
+def parseArgumentTupleAndReturnSlapgridObject(*argument_tuple):
+  """Returns a new instance of slapgrid.Slapgrid created with argument+config parameters.
+     Also returns the pidfile path, and configures logger.
+  """
+  args = parse_arguments(*argument_tuple)
+
+  configp = ConfigParser.SafeConfigParser()
+  configp.readfp(args.configuration_file)
+
+  options = merged_options(args, configp)
+
+  logger = setup_logger(options)
+
+  check_missing_parameters(options)
+  check_missing_files(options)
+
+  random_delay(options, logger=logger)
+
+  slapgrid_object = create_slapgrid_object(options, logger=logger)
+
+  return slapgrid_object, options.get('pidfile')
+
+
+def realRun(argument_tuple, method):
+  slapgrid_object, pidfile = parseArgumentTupleAndReturnSlapgridObject(*argument_tuple)
+  if pidfile:
+    setRunning(logger=slapgrid_object.logger, pidfile=pidfile)
+  try:
+    return getattr(slapgrid_object, method)()
+  finally:
+    if pidfile:
+      setFinished(pidfile)
+
+
+def runSoftwareRelease(*argument_tuple):
+  """Hook for entry point to process Software Releases"""
+  sys.exit(realRun(argument_tuple, 'processSoftwareReleaseList'))
+
+
+def runComputerPartition(*argument_tuple):
+  """Hook for entry point to process Computer Partitions"""
+  sys.exit(realRun(argument_tuple, 'processComputerPartitionList'))
+
+
+def runUsageReport(*argument_tuple):
+  """Hook for entry point to process Usage Reports"""
+  sys.exit(realRun(argument_tuple, 'agregateAndSendUsage'))

slapos/cli_legacy/supply.py

+# -*- coding: utf-8 -*-
+
+import argparse
+
+from slapos.client import ClientConfig, init, do_supply
+from slapos.cli_legacy.util import get_config_parser
+
+
+def supply():
+  """
+  Run when invoking slapos supply. Mostly argument parsing.
+  """
+  ap = argparse.ArgumentParser()
+  ap.add_argument('configuration_file',
+                  help='SlapOS configuration file')
+  ap.add_argument('software_url',
+                  help='Your software url')
+  ap.add_argument('node',
+                  help='Target node')
+  args = ap.parse_args()
+
+  configp = get_config_parser(args.configuration_file)
+  conf = ClientConfig(args, configp)
+  local = init(conf)
+  do_supply(args.software_url, args.node, local)

slapos/cli_legacy/svcbackend.py

+# -*- coding: utf-8 -*-
+# vim: set et sts=2:
+##############################################################################
+#
+# Copyright (c) 2010, 2011, 2012 Vifib SARL and Contributors.
+# All Rights Reserved.
+#
+# WARNING: This program as such is intended to be used by professional
+# programmers who take the whole responsibility of assessing all potential
+# consequences resulting from its eventual inadequacies and bugs
+# End users who are looking for a ready-to-use solution with commercial
+# guarantees and support are strongly advised to contract a Free Software
+# Service Company
+#
+# This program is Free Software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License
+# as published by the Free Software Foundation; either version 3
+# of the License, or (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA  02111-1307, USA.
+#
+##############################################################################
+
+import logging
+import os
+from optparse import OptionParser
+import ConfigParser
+
+from slapos.grid.svcbackend import launchSupervisord
+
+
+def getOptionDict(*argument_tuple):
+  usage = """
+Typical usage:
+ * %prog CONFIGURATION_FILE [arguments passed to supervisor]
+
+""".strip()
+
+  parser = OptionParser(usage=usage)
+
+  # Parses arguments
+  if argument_tuple:
+    (argument_option_instance, argument_list) = parser.parse_args(list(argument_tuple))
+  else:
+    # No arguments given to entry point : we parse sys.argv.
+    (argument_option_instance, argument_list) = parser.parse_args()
+
+  if not argument_list:
+    parser.error("Configuration file is obligatory. Consult documentation by calling with -h.")
+  configuration_file = argument_list[0]
+  if not os.path.exists(configuration_file):
+    parser.error("Could not read configuration file : %s" % configuration_file)
+
+  slapgrid_configuration = ConfigParser.SafeConfigParser()
+  slapgrid_configuration.read(configuration_file)
+
+  # Merges the two dictionnaries
+  option_dict = dict(slapgrid_configuration.items("slapos"))
+  # Supervisord configuration location
+  option_dict.setdefault('supervisord_configuration_path',
+                         os.path.join(option_dict['instance_root'], 'etc', 'supervisord.conf'))
+  # Supervisord socket
+  option_dict.setdefault('supervisord_socket',
+                         os.path.join(option_dict['instance_root'], 'supervisord.socket'))
+  return option_dict, argument_list[1:]
+
+
+def supervisorctl(*argument_tuple):
+  logger = logging.getLogger('SVCBackend')
+  logger.setLevel(logging.INFO)
+  handler = logging.StreamHandler()
+  logger.addHandler(handler)
+
+  option_dict, args = getOptionDict(*argument_tuple)
+  import supervisor.supervisorctl
+  launchSupervisord(socket=option_dict['supervisord_socket'],
+                    configuration_file=option_dict['supervisord_configuration_path'],
+                    logger=logger)
+  supervisor.supervisorctl.main(args=['-c', option_dict['supervisord_configuration_path']] + args)
+
+
+def supervisord(*argument_tuple):
+  logger = logging.getLogger('SVCBackend')
+  logger.setLevel(logging.INFO)
+  handler = logging.StreamHandler()
+  logger.addHandler(handler)
+
+  option_dict, _ = getOptionDict(*argument_tuple)
+  launchSupervisord(socket=option_dict['supervisord_socket'],
+                    configuration_file=option_dict['supervisord_configuration_path'],
+                    logger=logger)

slapos/cli_legacy/util.py

+# -*- coding: utf-8 -*-
+
+import ConfigParser
+import os
+
+
+def get_config_parser(path):
+    configp = ConfigParser.SafeConfigParser()
+    path = os.path.expanduser(path)
+    if not os.path.isfile(path):
+        raise OSError('Specified configuration file %s does not exist. Exiting.' % path)
+    configp.read(path)
+    return configp

slapos/client.py

@@ -27,121 +27,37 @@
 #
 ##############################################################################
 
-import argparse
+import atexit
 import ConfigParser
-import pprint
-from optparse import OptionParser, Option
 import os
-from slapos.slap import ResourceNotReady
-import slapos.slap.slap
-import sys
-import atexit
-
-class Parser(OptionParser):
-  """
-  Parse all arguments.
-  """
-  def __init__(self, usage=None, version=None):
-    """
-    Initialize all options possibles.
-    """
-    OptionParser.__init__(self, usage=usage, version=version,
-                          option_list=[
-        Option("-u", "--master_url",
-               default=None,
-               action="store",
-               help="Url of SlapOS Master to use."),
-        Option("-k", "--key_file",
-              action="store",
-              help="SSL Authorisation key file."),
-        Option("-c", "--cert_file",
-            action="store",
-            help="SSL Authorisation certificate file.")
-    ])
-
-  def check_args(self):
-    """
-    Check arguments
-    """
-    (options, args) = self.parse_args()
-    if len(args) == 0:
-      self.error("Incorrect number of arguments")
-    elif not os.path.isfile(args[0]):
-      self.error("%s: Not found or not a regular file." % args[0])
-
-    # Return options and only first element of args since there is only one.
-    return options, args[0]
-
+import pprint
 
-def argToDict(element):
-  """
-  convert a table of string 'key=value' to dict
-  """
-  if element is not None:
-    element_dict = dict([arg.split('=') for arg in element])
-  return element_dict
+import slapos.slap.slap
+from slapos.slap import ResourceNotReady
 
-def check_request_args():
-  """
-  Parser for request
-  """
-  parser = argparse.ArgumentParser()
-  parser.add_argument("configuration_file",
-                      help="SlapOS configuration file.")
-  parser.add_argument("reference",
-                      help="Your instance reference")
-  parser.add_argument("software_url",
-                      help="Your software url")
-  parser.add_argument("--node",
-                      nargs = '*',
-                      help = "Node request option "
-                      "'option1=value1 option2=value2'")
-  parser.add_argument("--type",
-                      type = str,
-                      help = "Define software type to be requested")
-  parser.add_argument("--slave",
-                      action = "store_true", default=False,
-                      help = "Ask for a slave instance")
-  parser.add_argument("--configuration",
-                      nargs = '*',
-                      help = "Give your configuration "
-                      "'option1=value1 option2=value2'")
-  args = parser.parse_args()
-  # Convert to dict
-  if args.configuration is not None:
-    args.configuration = argToDict(args.configuration)
-  if args.node is not None:
-    args.node = argToDict(args.node)
-  return args
 
+class ClientConfig(object):
+  state = None
 
-class Config:
-  def __init__(self, option_dict, configuration_file_path=None):
+  def __init__(self, args, configp=None):
+    # XXX configp cannot possibly be optional
     """
     Set options given by parameters.
     """
     # Set options parameters
-    for option, value in option_dict.__dict__.items():
-      setattr(self, option, value)
+    for key, value in args.__dict__.items():
+      setattr(self, key, value)
 
-    # Load configuration file
-    configuration_parser = ConfigParser.SafeConfigParser()
-    if configuration_file_path:
-      configuration_file_path = os.path.expanduser(configuration_file_path)
-      if not os.path.isfile(configuration_file_path):
-        raise OSError('Specified configuration file %s does not exist.'
-            ' Exiting.' % configuration_file_path)
-      configuration_parser.read(configuration_file_path)
     # Merges the arguments and configuration
     try:
-      configuration_dict = dict(configuration_parser.items('slapconsole'))
+      configuration_dict = dict(configp.items('slapconsole'))
     except ConfigParser.NoSectionError:
       pass
     else:
       for key in configuration_dict:
         if not getattr(self, key, None):
           setattr(self, key, configuration_dict[key])
-    configuration_dict = dict(configuration_parser.items('slapos'))
+    configuration_dict = dict(configp.items('slapos'))
     master_url = configuration_dict.get('master_url', None)
     # Backward compatibility, if no key and certificate given in option
     # take one from slapos configuration
@@ -158,67 +74,73 @@ class Config:
     else:
       setattr(self, 'master_url', master_url)
 
-def init(config):
+    if self.key_file:
+        self.key_file = os.path.expanduser(self.key_file)
+
+    if self.cert_file:
+        self.cert_file = os.path.expanduser(self.cert_file)
+
+def init(conf):
   """Initialize Slap instance, connect to server and create
   aliases to common software releases"""
+  # XXX check certificate and key existence
   slap = slapos.slap.slap()
-  slap.initializeConnection(config.master_url,
-      key_file=config.key_file, cert_file=config.cert_file)
+  slap.initializeConnection(conf.master_url,
+      key_file=conf.key_file, cert_file=conf.cert_file)
   local = globals().copy()
   local['slap'] = slap
   # Create aliases as global variables
   try:
-    alias = config.alias.split('\n')
+    alias = conf.alias.split('\n')
   except AttributeError:
     alias = []
   software_list = []
   for software in alias:
-    if software is not '':
+    if software:
       name, url = software.split(' ')
       software_list.append(name)
       local[name] = url
   # Create global variable too see available aliases
   local['software_list'] = software_list
   # Create global shortcut functions to request instance and software
+
   def shorthandRequest(*args, **kwargs):
     return slap.registerOpenOrder().request(*args, **kwargs)
+
   def shorthandSupply(*args, **kwargs):
     return slap.registerSupply().supply(*args, **kwargs)
+
   local['request'] = shorthandRequest
   local['supply'] = shorthandSupply
 
   return local
 
-def request():
-  """Run when invoking slapos request. Request an instance."""
-  # Parse arguments and inititate needed parameters
-  # XXX-Cedric: move argument parsing to main entry point
-  options = check_request_args()
-  config = Config(options, options.configuration_file)
-  local = init(config)
-  # Request instance
-  print("Requesting %s..." % config.reference)
-  if config.software_url in local:
-    config.software_url = local[config.software_url]
+
+def do_request(conf, local):
+  print("Requesting %s..." % conf.reference)
+  if conf.software_url in local:
+    conf.software_url = local[conf.software_url]
   try:
     partition = local['slap'].registerOpenOrder().request(
-      software_release = config.software_url,
-      partition_reference = config.reference,
-      partition_parameter_kw = config.configuration,
-      software_type = config.type,
-      filter_kw = config.node,
-      shared = config.slave
+      software_release = conf.software_url,
+      partition_reference = conf.reference,
+      partition_parameter_kw = conf.parameters,
+      software_type = conf.type,
+      filter_kw = conf.node,
+      state = conf.state,
+      shared = conf.slave
     )
     print "Instance requested.\nState is : %s." % partition.getState()
     print "Connection parameters of instance are:"
     pprint.pprint(partition.getConnectionParameterDict())
     print "You can rerun command to get up-to-date informations."
   except ResourceNotReady:
-    print("Instance requested. Master is provisionning it. Please rerun in a "
+    print("Instance requested. Master is provisioning it. Please rerun in a "
         "couple of minutes to get connection informations.")
     exit(2)
 
-def _supply(software_url, computer_id, local, remove=False):
+
+def do_supply(software_url, computer_id, local, remove=False):
   """
   Request installation of Software Release
   'software_url' on computer 'computer_id'.
@@ -243,58 +165,12 @@ def _supply(software_url, computer_id, local, remove=False):
   )
   print 'Done.'
 
-def supply():
-  """
-  Run when invoking slapos supply. Mostly argument parsing.
-  """
-  # XXX-Cedric: move argument parsing to main entry point
-  parser = argparse.ArgumentParser()
-  parser.add_argument("configuration_file",
-                      help="SlapOS configuration file")
-  parser.add_argument("software_url",
-                      help="Your software url")
-  parser.add_argument("node",
-                      help="Target node")
-  args = parser.parse_args()
-
-  config = Config(args, args.configuration_file)
-  _supply(args.software_url, args.node, init(config))
 
-def remove():
-  """
-  Run when invoking slapos remove. Mostly argument parsing.
-  """
-  # XXX-Cedric: move argument parsing to main entry point
-  parser = argparse.ArgumentParser()
-  parser.add_argument("configuration_file",
-                      help="SlapOS configuration file.")
-  parser.add_argument("software_url",
-                      help="Your software url")
-  parser.add_argument("node",
-                      help="Target node")
-  args = parser.parse_args()
+def do_remove(software_url, node, local):
+  do_supply(software_url, node, local, remove=True)
 
-  config = Config(args, args.configuration_file)
-  _supply(args.software_url, args.node, init(config), remove=True)
-
-
-def slapconsole():
-  """Ran when invoking slapconsole"""
-  # Parse arguments
-  usage = """usage: %s [options] CONFIGURATION_FILE
-slapconsole allows you interact with slap API. You can play with the global
-"slap" object and with the global "request" method.
-
-examples :
-  >>> # Request instance
-  >>> request(kvm, "myuniquekvm")
-  >>> # Request software installation on owned computer
-  >>> supply(kvm, "mycomputer")
-  >>> # Fetch instance informations on already launched instance
-  >>> request(kvm, "myuniquekvm").getConnectionParameter("url")""" % sys.argv[0]
-  config = Config(*Parser(usage=usage).check_args())
-  local = init(config)
 
+def do_console(local):
   # try to enable readline with completion and history
   try:
     import readline
@@ -316,4 +192,3 @@ examples :
     atexit.register(save_history)
 
   __import__("code").interact(banner="", local=local)
-

slapos/format.py

@@ -28,8 +28,6 @@
 #
 ##############################################################################
 
-from optparse import OptionParser, Option
-from xml_marshaller import xml_marshaller
 import ConfigParser
 import errno
 import fcntl
@@ -41,7 +39,6 @@ import netifaces
 import os
 import pwd
 import random
-import slapos.slap as slap
 import shutil
 import socket
 import struct
@@ -53,12 +50,10 @@ import traceback
 import zipfile
 
 import lxml.etree
-from slapos.version import version
-
+import xml_marshaller.xml_marshaller
 
-# set up logging
-logger = logging.getLogger("slapformat")
-logger.setLevel(logging.INFO)
+from slapos.util import mkdir_p
+import slapos.slap as slap
 
 
 def prettify_xml(xml):
@@ -66,17 +61,14 @@ def prettify_xml(xml):
   return lxml.etree.tostring(root, pretty_print=True)
 
 
-from slapos.util import mkdir_p
-
 class OS(object):
   """Wrap parts of the 'os' module to provide logging of performed actions."""
 
   _os = os
 
-  def __init__(self, config):
-    self._dry_run = config.dry_run
-    self._verbose = config.verbose
-    self._logger = config.logger
+  def __init__(self, conf):
+    self._dry_run = conf.dry_run
+    self._logger = conf.logger
     add = self._addWrapper
     add('chown')
     add('chmod')
@@ -85,7 +77,6 @@ class OS(object):
 
   def _addWrapper(self, name):
     def wrapper(*args, **kw):
-      if self._verbose:
       arg_list = [repr(x) for x in args] + [
               '%s=%r' % (x, y) for x, y in kw.iteritems()
               ]
@@ -161,36 +152,35 @@ def netmaskToPrefixIPv6(netmask):
           netaddr.strategy.ipv6.str_to_int(netmask)]
 
 
-def _getDict(instance):
+def _getDict(obj):
   """
-  Serialize an object instance into dictionaries. List and dict will remains
+  Serialize an object into dictionaries. List and dict will remains
   the same, basic type too. But encapsulated object will be returned as dict.
   Set, collections and other aren't handle for now.
 
   Args:
-    instance: an object of any type.
+    obj: an object of any type.
 
   Returns:
     A dictionary if the given object wasn't a list, a list otherwise.
   """
-  if isinstance(instance, list):
-    return [_getDict(item) for item in instance]
-
-  elif isinstance(instance, dict):
-    result = {}
-    for key in instance:
-      result[key] = _getDict(instance[key])
-    return result
+  if isinstance(obj, list):
+    return [_getDict(item) for item in obj]
 
+  if isinstance(obj, dict):
+    dikt = obj
   else:
     try:
-      dikt = instance.__dict__
+      dikt = obj.__dict__
     except AttributeError:
-      return instance
-    result = {}
-    for key, value in dikt.iteritems():
-      result[key] = _getDict(value)
-    return result
+      return obj
+
+  return {
+          key: _getDict(value)
+          for key, value in dikt.iteritems()
+          # do not attempt to serialize logger: it is both useless and recursive.
+          if not isinstance(value, logging.Logger)
+          }
 
 
 class Computer(object):
@@ -248,30 +238,30 @@ class Computer(object):
     # Can't find address
     raise NoAddressOnInterface('No valid IPv6 found on %s.' % self.interface.name)
 
-  def send(self, config):
+  def send(self, conf):
     """
     Send a marshalled dictionary of the computer object serialized via_getDict.
     """
 
     slap_instance = slap.slap()
     connection_dict = {}
-    if config.key_file and config.cert_file:
-      connection_dict['key_file'] = config.key_file
-      connection_dict['cert_file'] = config.cert_file
-    slap_instance.initializeConnection(config.master_url,
+    if conf.key_file and conf.cert_file:
+      connection_dict['key_file'] = conf.key_file
+      connection_dict['cert_file'] = conf.cert_file
+    slap_instance.initializeConnection(conf.master_url,
                                        **connection_dict)
     slap_computer = slap_instance.registerComputer(self.reference)
 
-    if config.dry_run:
+    if conf.dry_run:
       return
     try:
-      slap_computer.updateConfiguration(xml_marshaller.dumps(_getDict(self)))
+      slap_computer.updateConfiguration(xml_marshaller.xml_marshaller.dumps(_getDict(self)))
     except slap.NotFoundError as error:
       raise slap.NotFoundError("%s\nERROR : This SlapOS node is not recognised by "
           "SlapOS Master. Please make sure computer_id of slapos.cfg looks "
           "like 'COMP-123' and is correct.\nError is : 404 Not Found." % error)
 
-  def dump(self, path_to_xml, path_to_json):
+  def dump(self, path_to_xml, path_to_json, logger):
     """
     Dump the computer object to an xml file via xml_marshaller.
 
@@ -286,7 +276,7 @@ class Computer(object):
       with open(path_to_json, 'wb') as fout:
         fout.write(json.dumps(computer_dict, sort_keys=True, indent=2))
 
-    new_xml = xml_marshaller.dumps(computer_dict)
+    new_xml = xml_marshaller.xml_marshaller.dumps(computer_dict)
     new_pretty_xml = prettify_xml(new_xml)
 
     path_to_archive = path_to_xml + '.zip'
@@ -340,7 +330,7 @@ class Computer(object):
       A Computer object.
     """
 
-    dumped_dict = xml_marshaller.loads(open(path_to_xml).read())
+    dumped_dict = xml_marshaller.xml_marshaller.loads(open(path_to_xml).read())
 
     # Reconstructing the computer object from the xml
     computer = Computer(
@@ -684,12 +674,13 @@ class Tap(object):
 class Interface(object):
   """Represent a network interface on the system"""
 
-  def __init__(self, name, ipv4_local_network, ipv6_interface=None):
+  def __init__(self, logger, name, ipv4_local_network, ipv6_interface=None):
     """
     Attributes:
         name: String, the name of the interface
     """
 
+    self.logger = logger
     self.name = str(name)
     self.ipv4_local_network = ipv4_local_network
     self.ipv6_interface = ipv6_interface
@@ -853,7 +844,7 @@ class Interface(object):
       if self._addSystemAddress(addr, netmask, False):
         return dict(addr=addr, netmask=netmask)
       else:
-        logger.warning('Impossible to add old local IPv4 %s. Generating '
+        self.logger.warning('Impossible to add old local IPv4 %s. Generating '
             'new IPv4 address.' % addr)
         return self._generateRandomIPv4Address(netmask)
     else:
@@ -911,7 +902,7 @@ class Interface(object):
             # succeed, return it
             return dict(addr=addr, netmask=netmask)
           else:
-            logger.warning('Impossible to add old public IPv6 %s. '
+            self.logger.warning('Impossible to add old public IPv6 %s. '
                 'Generating new IPv6 address.' % addr)
 
     # Try 10 times to add address, raise in case if not possible
@@ -931,76 +922,8 @@ class Interface(object):
     raise AddressGenerationError(addr)
 
 
-class Parser(OptionParser):
-  """
-  Parse all arguments.
-  """
-  def __init__(self, usage=None, version=version):
-    """
-    Initialize all options possibles.
-    """
-    OptionParser.__init__(self, usage=usage, version=version,
-                          option_list=[
-      Option("-x", "--computer_xml",
-             help="Path to file with computer's XML. If does not exists, "
-                  "will be created",
-             default=None,
-             type=str),
-      Option("--computer_json",
-             help="Path to a JSON version of the computer's XML (for development only).",
-             default=None,
-             type=str),
-      Option("-l", "--log_file",
-             help="The path to the log file used by the script.",
-             type=str),
-      Option("-i", "--input_definition_file",
-             help="Path to file to read definition of computer instead of "
-             "declaration. Using definition file allows to disable "
-             "'discovery' of machine services and allows to define computer "
-             "configuration in fully controlled manner.",
-             type=str),
-      Option("-o", "--output_definition_file",
-             help="Path to file to write definition of computer from "
-             "declaration.",
-             type=str),
-      Option("-n", "--dry_run",
-             help="Don't actually do anything.",
-             default=False,
-             action="store_true"),
-      Option("-v", "--verbose",
-             default=False,
-             action="store_true",
-             help="Verbose output."),
-      Option("-c", "--console",
-             default=False,
-             action="store_true",
-             help="Console output."),
-      Option('--alter_user', choices=['True', 'False'],
-        help="Shall slapformat alter user database [default: True]"),
-      Option('--alter_network', choices=['True', 'False'],
-        help="Shall slapformat alter network configuration [default: True]"),
-      Option('--now',
-             help="Launch slapformat without delay",
-             default=False,
-             action="store_true"),
-      ])
-
-  def check_args(self, args):
-    """
-    Check arguments
-    """
-    if args:
-      (options, args) = self.parse_args(list(args))
-    else:
-      (options, args) = self.parse_args()
-    if len(args) != 1:
-      self.error("Incorrect number of arguments")
-    return options, args[0]
-
-
-
-def parse_computer_definition(config, definition_path):
-  config.logger.info('Using definition file %r' % definition_path)
+def parse_computer_definition(conf, definition_path):
+  conf.logger.info('Using definition file %r' % definition_path)
   computer_definition = ConfigParser.RawConfigParser({
     'software_user': 'slapsoft',
   })
@@ -1010,20 +933,20 @@ def parse_computer_definition(config, definition_path):
   netmask = None
   if computer_definition.has_option('computer', 'address'):
     address, netmask = computer_definition.get('computer', 'address').split('/')
-  if config.alter_network and config.interface_name is not None \
-      and config.ipv4_local_network is not None:
-    interface = Interface(config.interface_name, config.ipv4_local_network,
-      config.ipv6_interface)
+  if conf.alter_network and conf.interface_name is not None \
+      and conf.ipv4_local_network is not None:
+    interface = Interface(conf.logger, conf.interface_name, conf.ipv4_local_network,
+      conf.ipv6_interface)
   computer = Computer(
-      reference=config.computer_id,
+      reference=conf.computer_id,
       interface=interface,
       addr=address,
       netmask=netmask,
-      ipv6_interface=config.ipv6_interface,
+      ipv6_interface=conf.ipv6_interface,
       software_user=computer_definition.get('computer', 'software_user'),
     )
   partition_list = []
-  for partition_number in range(int(config.partition_amount)):
+  for partition_number in range(int(conf.partition_amount)):
     section = 'partition_%s' % partition_number
     user = User(computer_definition.get(section, 'user'))
     address_list = []
@@ -1032,7 +955,7 @@ def parse_computer_definition(config, definition_path):
       address_list.append(dict(addr=address, netmask=netmask))
     tap = Tap(computer_definition.get(section, 'network_interface'))
     partition = Partition(reference=computer_definition.get(section, 'pathname'),
-                          path=os.path.join(config.instance_root,
+                          path=os.path.join(conf.instance_root,
                                             computer_definition.get(section, 'pathname')),
                           user=user,
                           address_list=address_list,
@@ -1042,48 +965,48 @@ def parse_computer_definition(config, definition_path):
   return computer
 
 
-def parse_computer_xml(config, xml_path):
+def parse_computer_xml(conf, xml_path):
   if os.path.exists(xml_path):
-    config.logger.info('Loading previous computer data from %r' % xml_path)
+    conf.logger.info('Loading previous computer data from %r' % xml_path)
     computer = Computer.load(xml_path,
-                             reference=config.computer_id,
-                             ipv6_interface=config.ipv6_interface)
+                             reference=conf.computer_id,
+                             ipv6_interface=conf.ipv6_interface)
     # Connect to the interface defined by the configuration
-    computer.interface = Interface(config.interface_name, config.ipv4_local_network,
-        config.ipv6_interface)
+    computer.interface = Interface(conf.logger, conf.interface_name, conf.ipv4_local_network,
+        conf.ipv6_interface)
   else:
     # If no pre-existent configuration found, create a new computer object
-    config.logger.warning('Creating new data computer with id %r' % config.computer_id)
+    conf.logger.warning('Creating new data computer with id %r' % conf.computer_id)
     computer = Computer(
-      reference=config.computer_id,
-      interface=Interface(config.interface_name, config.ipv4_local_network,
-        config.ipv6_interface),
+      reference=conf.computer_id,
+      interface=Interface(conf.logger, conf.interface_name, conf.ipv4_local_network,
+        conf.ipv6_interface),
       addr=None,
       netmask=None,
-      ipv6_interface=config.ipv6_interface,
-      software_user=config.software_user,
+      ipv6_interface=conf.ipv6_interface,
+      software_user=conf.software_user,
     )
 
-  partition_amount = int(config.partition_amount)
+  partition_amount = int(conf.partition_amount)
   existing_partition_amount = len(computer.partition_list)
   if existing_partition_amount > partition_amount:
     raise ValueError('Requested amount of computer partitions (%s) is lower '
         'then already configured (%s), cannot continue' % (partition_amount,
           len(computer.partition_list)))
 
-  config.logger.info('Adding %s new partitions' %
-      (partition_amount-existing_partition_amount))
+  conf.logger.info('Adding %s new partitions' %
+      (partition_amount - existing_partition_amount))
   for nb_iter in range(existing_partition_amount, partition_amount):
     # add new ones
-    user = User("%s%s" % (config.user_base_name, nb_iter))
+    user = User("%s%s" % (conf.user_base_name, nb_iter))
 
-    tap = Tap("%s%s" % (config.tap_base_name, nb_iter))
+    tap = Tap("%s%s" % (conf.tap_base_name, nb_iter))
 
-    path = os.path.join(config.instance_root, "%s%s" % (
-                         config.partition_base_name, nb_iter))
+    path = os.path.join(conf.instance_root, "%s%s" % (
+                         conf.partition_base_name, nb_iter))
     computer.partition_list.append(
       Partition(
-        reference="%s%s" % (config.partition_base_name, nb_iter),
+        reference="%s%s" % (conf.partition_base_name, nb_iter),
         path=path,
         user=user,
         address_list=None,
@@ -1093,7 +1016,7 @@ def parse_computer_xml(config, xml_path):
   return computer
 
 
-def write_computer_definition(config, computer):
+def write_computer_definition(conf, computer):
   computer_definition = ConfigParser.RawConfigParser()
   computer_definition.add_section('computer')
   if computer.address is not None and computer.netmask is not None:
@@ -1109,44 +1032,58 @@ def write_computer_definition(config, computer):
     computer_definition.set(section, 'user', partition.user.name)
     computer_definition.set(section, 'network_interface', partition.tap.name)
     computer_definition.set(section, 'pathname', partition.reference)
-  computer_definition.write(open(config.output_definition_file, 'w'))
-  config.logger.info('Stored computer definition in %r' % config.output_definition_file)
+  computer_definition.write(open(conf.output_definition_file, 'w'))
+  conf.logger.info('Stored computer definition in %r' % conf.output_definition_file)
+
+
+def random_delay(conf):
+  # Add delay between 0 and 1 hour
+  # XXX should be the contrary: now by default, and cron should have
+  # --maximal-delay=3600
+  if not conf.now:
+    duration = float(60 * 60) * random.random()
+    print("Sleeping for %s seconds. To disable this feature, " \
+                    "use with --now parameter in manual." % duration)
+    time.sleep(duration)
 
 
-def run(config):
-  if config.input_definition_file:
-    computer = parse_computer_definition(config, config.input_definition_file)
+def do_format(conf):
+  random_delay(conf)
+
+  if conf.input_definition_file:
+    computer = parse_computer_definition(conf, conf.input_definition_file)
   else:
     # no definition file, figure out computer
-    computer = parse_computer_xml(config, config.computer_xml)
+    computer = parse_computer_xml(conf, conf.computer_xml)
 
-  computer.instance_root = config.instance_root
-  computer.software_root = config.software_root
-  config.logger.info('Updating computer')
-  address = computer.getAddress(config.create_tap)
+  computer.instance_root = conf.instance_root
+  computer.software_root = conf.software_root
+  conf.logger.info('Updating computer')
+  address = computer.getAddress(conf.create_tap)
   computer.address = address['addr']
   computer.netmask = address['netmask']
 
-  if config.output_definition_file:
-    write_computer_definition(config, computer)
+  if conf.output_definition_file:
+    write_computer_definition(conf, computer)
 
-  computer.construct(alter_user=config.alter_user,
-                     alter_network=config.alter_network,
-                     create_tap=config.create_tap)
+  computer.construct(alter_user=conf.alter_user,
+                     alter_network=conf.alter_network,
+                     create_tap=conf.create_tap)
 
-  if getattr(config, 'certificate_repository_path', None):
-    mkdir_p(config.certificate_repository_path, mode=0o700)
+  if getattr(conf, 'certificate_repository_path', None):
+    mkdir_p(conf.certificate_repository_path, mode=0o700)
 
   # Dumping and sending to the erp5 the current configuration
-  if not config.dry_run:
-    computer.dump(path_to_xml=config.computer_xml,
-                  path_to_json=config.computer_json)
-  config.logger.info('Posting information to %r' % config.master_url)
-  computer.send(config)
-  config.logger.info('slapformat successfully prepared computer.')
+  if not conf.dry_run:
+    computer.dump(path_to_xml=conf.computer_xml,
+                  path_to_json=conf.computer_json,
+                  logger=conf.logger)
+  conf.logger.info('Posting information to %r' % conf.master_url)
+  computer.send(conf)
+  conf.logger.info('slapformat successfully prepared computer.')
 
 
-class Config(object):
+class FormatConfig(object):
   key_file = None
   cert_file = None
   alter_network = None
@@ -1155,14 +1092,14 @@ class Config(object):
   computer_xml = None
   computer_json = None
   input_definition_file = None
-  logger = None
   log_file = None
   output_definition_file = None
-  verbose = None
   dry_run = None
-  console = None
   software_user = None
 
+  def __init__(self, logger):
+    self.logger = logger
+
   @staticmethod
   def checkRequiredBinary(binary_list):
     missing_binary_list = []
@@ -1179,33 +1116,26 @@ class Config(object):
       raise UsageError('Some required binaries are missing or not '
           'functional: %s' % (','.join(missing_binary_list), ))
 
-  def setConfig(self, option_dict, configuration_file_path):
+  def mergeConfig(self, args, configp):
     """
     Set options given by parameters.
+    Must be executed before setting up the logger.
     """
     self.key_file = None
     self.cert_file = None
 
-    # set up logging
-    # XXX-Cedric: change code to use global logger
-    self.logger = logger
-
-    # Set options parameters
-    for option, value in option_dict.__dict__.items():
-      setattr(self, option, value)
-
-    # Load configuration file
-    configuration_parser = ConfigParser.SafeConfigParser()
-    if configuration_parser.read(configuration_file_path) != [configuration_file_path]:
-      raise UsageError('Cannot find or parse configuration file: %s' % configuration_file_path)
+    # Set argument parameters
+    for key, value in args.__dict__.items():
+      setattr(self, key, value)
 
     # Merges the arguments and configuration
     for section in ("slapformat", "slapos"):
-      configuration_dict = dict(configuration_parser.items(section))
+      configuration_dict = dict(configp.items(section))
       for key in configuration_dict:
         if not getattr(self, key, None):
           setattr(self, key, configuration_dict[key])
 
+  def setConfig(self):
     # setup some nones
     for parameter in ['interface_name', 'partition_base_name', 'user_base_name',
         'tap_base_name', 'ipv4_local_network', 'ipv6_interface']:
@@ -1234,10 +1164,6 @@ class Config(object):
     if self.create_tap is None:
       self.create_tap = True
 
-    # Configure logging
-    if self.console:
-      self.logger.addHandler(logging.StreamHandler())
-
     # Convert strings to booleans
     for o in ['alter_network', 'alter_user', 'create_tap']:
       attr = getattr(self, o)
@@ -1275,22 +1201,9 @@ class Config(object):
     if root_needed and os.getuid() != 0:
       message = "Root rights are needed"
       self.logger.error(message)
-      sys.stderr.write(message+'\n')
+      sys.stderr.write(message + '\n')
       sys.exit()
 
-    if self.log_file:
-      if not os.path.isdir(os.path.dirname(self.log_file)):
-        # fallback to console only if directory for logs does not exists and
-        # continue to run
-        raise ValueError('Please create directory %r to store %r log file' % (
-          os.path.dirname(self.log_file), self.log_file))
-      else:
-        file_handler = logging.FileHandler(self.log_file)
-        file_handler.setFormatter(logging.Formatter("%(asctime)s - "
-          "%(name)s - %(levelname)s - %(message)s"))
-        self.logger.addHandler(file_handler)
-        self.logger.info('Configured logging to file %r' % self.log_file)
-
     # Check mandatory options
     for parameter in ('computer_id', 'instance_root', 'master_url',
                       'software_root', 'computer_xml'):
@@ -1307,9 +1220,6 @@ class Config(object):
           sys.exit(1)
 
     self.logger.info("Started.")
-    if self.verbose:
-      self.logger.setLevel(logging.DEBUG)
-      self.logger.debug("Verbose mode enabled.")
     if self.dry_run:
       self.logger.info("Dry-run mode enabled.")
     if self.create_tap:
@@ -1325,22 +1235,22 @@ class Config(object):
       self.output_definition_file = os.path.abspath(self.output_definition_file)
 
 
-
-def tracing_monkeypatch(config):
+def tracing_monkeypatch(conf):
   """Substitute os module and callAndRead function with tracing wrappers."""
   global os
   global callAndRead
 
   real_callAndRead = callAndRead
 
-  os = OS(config)
-  if config.dry_run:
+  os = OS(conf)
+  if conf.dry_run:
     def dry_callAndRead(argument_list, raise_on_error=True):
       if argument_list == ['brctl', 'show']:
         return real_callAndRead(argument_list, raise_on_error)
       else:
         return 0, ''
     callAndRead = dry_callAndRead
+
     def fake_getpwnam(user):
       class result(object):
         pw_uid = 12345
@@ -1350,40 +1260,7 @@ def tracing_monkeypatch(config):
   else:
     dry_callAndRead = real_callAndRead
 
-  if config.verbose:
   def logging_callAndRead(argument_list, raise_on_error=True):
-      config.logger.debug(' '.join(argument_list))
+    conf.logger.debug(' '.join(argument_list))
     return dry_callAndRead(argument_list, raise_on_error)
   callAndRead = logging_callAndRead
-
-
-def main(*args):
-  "Run default configuration."
-
-  # Parse arguments
-  usage = "usage: %s [options] CONFIGURATION_FILE" % sys.argv[0]
-  options, configuration_file_path = Parser(usage=usage).check_args(args)
-  config = Config()
-  try:
-    config.setConfig(options, configuration_file_path)
-  except UsageError as err:
-    sys.stderr.write(err.message + '\n')
-    sys.stderr.write("For help use --help\n")
-    sys.exit(1)
-
-  tracing_monkeypatch(config)
-
-  # Add delay between 0 and 1 hour
-  # XXX should be the contrary: now by default, and cron should have
-  # --maximal-delay=3600
-  if not config.now:
-    duration = float(60*60) * random.random()
-    print("Sleeping for %s seconds. To disable this feature, " \
-                    "use with --now parameter in manual." % duration)
-    time.sleep(duration)
-  try:
-    run(config)
-  except:
-    config.logger.exception('Uncaught exception:')
-    raise
-

slapos/grid/SlapObject.py

@@ -28,50 +28,59 @@
 #
 ##############################################################################
 
-import logging
 import os
-import shutil
-import subprocess
 import pkg_resources
 import pwd
+import shutil
 import stat
+import subprocess
 import tarfile
 import tempfile
 import textwrap
-import utils
 import xmlrpclib
 
 from supervisor import xmlrpc
+
+from slapos.grid.utils import (md5digest, getCleanEnvironment,
+                               SlapPopen, dropPrivileges, updateFile)
+from slapos.grid import utils # for mocked methods
 from slapos.slap.slap import NotFoundError
-from svcbackend import getSupervisorRPC
-from exception import BuildoutFailedError, WrongPermissionError, \
-    PathDoesNotExistError
-from networkcache import download_network_cached, upload_network_cached
-from watchdog import getWatchdogID
+from slapos.grid.svcbackend import getSupervisorRPC
+from slapos.grid.exception import (BuildoutFailedError, WrongPermissionError,
+                                   PathDoesNotExistError)
+from slapos.grid.networkcache import download_network_cached, upload_network_cached
+from slapos.grid.watchdog import getWatchdogID
 
-REQUIRED_COMPUTER_PARTITION_PERMISSION = '0750'
+REQUIRED_COMPUTER_PARTITION_PERMISSION = 0o750
 
 
 class Software(object):
   """This class is responsible for installing a software release"""
 
-  def __init__(self, url, software_root, buildout,
+  def __init__(self, url, software_root, buildout, logger,
                signature_private_key_file=None, signature_certificate_list=None,
                upload_cache_url=None, upload_dir_url=None, shacache_cert_file=None,
                shacache_key_file=None, shadir_cert_file=None, shadir_key_file=None,
                download_binary_cache_url=None, upload_binary_cache_url=None,
                download_binary_dir_url=None, upload_binary_dir_url=None,
-      download_from_binary_cache_url_blacklist = [],
-      upload_to_binary_cache_url_blacklist = []):
+               download_from_binary_cache_url_blacklist=None,
+               upload_to_binary_cache_url_blacklist=None):
     """Initialisation of class parameters
     """
+
+    if download_from_binary_cache_url_blacklist is None:
+      download_from_binary_cache_url_blacklist = []
+
+    if upload_to_binary_cache_url_blacklist is None:
+      upload_to_binary_cache_url_blacklist = []
+
     self.url = url
     self.software_root = software_root
-    self.software_url_hash = utils.getSoftwareUrlHash(self.url)
+    self.software_url_hash = md5digest(self.url)
     self.software_path = os.path.join(self.software_root,
                                       self.software_url_hash)
     self.buildout = buildout
-    self.logger = logging.getLogger('BuildoutManager')
+    self.logger = logger
     self.signature_private_key_file = signature_private_key_file
     self.signature_certificate_list = signature_certificate_list
     self.upload_cache_url = upload_cache_url
@@ -116,9 +125,8 @@ class Software(object):
       else:
         self._install_from_buildout()
         # Upload to binary cache if possible and allowed
-        if (self.software_root and self.url and self.software_url_hash \
-                               and self.upload_binary_cache_url \
-                               and self.upload_binary_dir_url):
+        if all([self.software_root, self.url, self.software_url_hash,
+                self.upload_binary_cache_url, self.upload_binary_dir_url]):
           blacklisted = False
           for url in self.upload_to_binary_cache_url_blacklist:
             if self.url.startswith(url):
@@ -136,8 +144,8 @@ class Software(object):
     it. If it fails, we notify the server.
     """
     root_stat_info = os.stat(self.software_root)
-    os.environ = utils.getCleanEnvironment(pwd.getpwuid(root_stat_info.st_uid
-      ).pw_dir)
+    os.environ = getCleanEnvironment(logger=self.logger,
+                                     home_path=pwd.getpwuid(root_stat_info.st_uid).pw_dir)
     if not os.path.isdir(self.software_path):
       os.mkdir(self.software_path)
     extends_cache = tempfile.mkdtemp()
@@ -145,40 +153,43 @@ class Software(object):
       # In case when running as root copy ownership, to simplify logic
       for path in [self.software_path, extends_cache]:
         path_stat_info = os.stat(path)
-        if root_stat_info.st_uid != path_stat_info.st_uid or\
-             root_stat_info.st_gid != path_stat_info.st_gid:
+        if (root_stat_info.st_uid != path_stat_info.st_uid or
+              root_stat_info.st_gid != path_stat_info.st_gid):
           os.chown(path, root_stat_info.st_uid,
                    root_stat_info.st_gid)
     try:
       buildout_parameter_list = [
           'buildout:extends-cache=%s' % extends_cache,
-        'buildout:directory=%s' % self.software_path,]
+          'buildout:directory=%s' % self.software_path
+      ]
 
-      if self.signature_private_key_file or \
-          self.upload_cache_url or \
-            self.upload_dir_url is not None:
+      if (self.signature_private_key_file or
+          self.upload_cache_url or
+          self.upload_dir_url):
         buildout_parameter_list.append('buildout:networkcache-section=networkcache')
-      for  buildout_option, value in (
+      for buildout_option, value in [
           ('%ssignature-private-key-file=%s', self.signature_private_key_file),
           ('%supload-cache-url=%s', self.upload_cache_url),
           ('%supload-dir-url=%s', self.upload_dir_url),
           ('%sshacache-cert-file=%s', self.shacache_cert_file),
           ('%sshacache-key-file=%s', self.shacache_key_file),
           ('%sshadir-cert-file=%s', self.shadir_cert_file),
-         ('%sshadir-key-file=%s', self.shadir_key_file),
-         ):
+          ('%sshadir-key-file=%s', self.shadir_key_file)
+      ]:
         if value:
-          buildout_parameter_list.append( \
-              buildout_option % ('networkcache:', value))
+          buildout_parameter_list.append(buildout_option % ('networkcache:', value))
 
       buildout_cfg = os.path.join(self.software_path, 'buildout.cfg')
       self.createProfileIfMissing(buildout_cfg, self.url)
 
       buildout_parameter_list.extend(['-c', buildout_cfg])
-      utils.bootstrapBuildout(self.software_path, self.buildout,
+      utils.bootstrapBuildout(path=self.software_path,
+                              buildout=self.buildout,
+                              logger=self.logger,
                               additional_buildout_parametr_list=buildout_parameter_list)
-      utils.launchBuildout(self.software_path,
-                     os.path.join(self.software_path, 'bin', 'buildout'),
+      utils.launchBuildout(path=self.software_path,
+                           buildout_binary=os.path.join(self.software_path, 'bin', 'buildout'),
+                           logger=self.logger,
                            additional_buildout_parametr_list=buildout_parameter_list)
     finally:
       shutil.rmtree(extends_cache)
@@ -226,7 +237,7 @@ class Software(object):
       if func == os.path.islink:
         os.unlink(path)
       else:
-        os.chmod(path, 0600)
+        os.chmod(path, 0o600)
         func(path)
     try:
       if os.path.exists(self.software_path):
@@ -235,10 +246,8 @@ class Software(object):
       else:
         self.logger.info('Path %r does not exists, no need to remove.' %
             self.software_path)
-    except IOError as error:
-      error_string = "I/O error while removing software (%s): %s" % (self.url,
-                                                                     error)
-      raise IOError(error_string)
+    except IOError as exc:
+      raise IOError("I/O error while removing software (%s): %s" % (self.url, exc))
 
 
 class Partition(object):
@@ -256,10 +265,12 @@ class Partition(object):
                server_url,
                software_release_url,
                buildout,
+               logger,
                certificate_repository_path=None,
                ):
     """Initialisation of class parameters"""
     self.buildout = buildout
+    self.logger = logger
     self.software_path = software_path
     self.instance_path = instance_path
     self.run_path = os.path.join(self.instance_path, 'etc', 'run')
@@ -268,7 +279,6 @@ class Partition(object):
         supervisord_partition_configuration_path
     self.supervisord_socket = supervisord_socket
     self.computer_partition = computer_partition
-    self.logger = logging.getLogger('Partition')
     self.computer_id = computer_id
     self.partition_id = partition_id
     self.server_url = server_url
@@ -284,19 +294,18 @@ class Partition(object):
       self._updateCertificate()
 
   def _updateCertificate(self):
-    if not os.path.exists(self.key_file) or \
-        not os.path.exists(self.cert_file):
+    if not os.path.exists(self.key_file) or not os.path.exists(self.cert_file):
       self.logger.info('Certificate and key not found, downloading to %r and '
           '%r' % (self.cert_file, self.key_file))
       try:
         partition_certificate = self.computer_partition.getCertificate()
       except NotFoundError:
-        raise NotFoundError('Partition %s is not known from SlapOS Master.' % \
+        raise NotFoundError('Partition %s is not known from SlapOS Master.' %
             self.partition_id)
       open(self.key_file, 'w').write(partition_certificate['key'])
       open(self.cert_file, 'w').write(partition_certificate['certificate'])
     for f in [self.key_file, self.cert_file]:
-      os.chmod(f, 0400)
+      os.chmod(f, 0o400)
       os.chown(f, *self.getUserGroupId())
 
   def getUserGroupId(self):
@@ -307,24 +316,24 @@ class Partition(object):
     return (uid, gid)
 
   def addServiceToGroup(self, partition_id,
-                        runner_list, path, extension = ''):
+                        runner_list, path, extension=''):
     uid, gid = self.getUserGroupId()
     program_partition_template = pkg_resources.resource_stream(__name__,
             'templates/program_partition_supervisord.conf.in').read()
     for runner in runner_list:
       self.partition_supervisor_configuration += '\n' + \
-          program_partition_template % dict(
-        program_id='_'.join([partition_id, runner]),
-        program_directory=self.instance_path,
-        program_command=os.path.join(path, runner),
-        program_name=runner+extension,
-        instance_path=self.instance_path,
-        user_id=uid,
-        group_id=gid,
+          program_partition_template % {
+                  'program_id': '_'.join([partition_id, runner]),
+                  'program_directory': self.instance_path,
+                  'program_command': os.path.join(path, runner),
+                  'program_name': runner + extension,
+                  'instance_path': self.instance_path,
+                  'user_id': uid,
+                  'group_id': gid,
                   # As supervisord has no environment to inherit, setup a minimalistic one
-        HOME=pwd.getpwuid(uid).pw_dir,
-        USER=pwd.getpwuid(uid).pw_name,
-      )
+                  'HOME': pwd.getpwuid(uid).pw_dir,
+                  'USER': pwd.getpwuid(uid).pw_name,
+                  }
 
   def updateSymlink(self, sr_symlink, software_path):
     if os.path.lexists(sr_symlink):
@@ -339,7 +348,7 @@ class Partition(object):
     """ Creates configuration file from template in software_path, then
     installs the software partition with the help of buildout
     """
-    self.logger.info("Installing Computer Partition %s..." \
+    self.logger.info("Installing Computer Partition %s..."
         % self.computer_partition.getId())
     # Checks existence and permissions of Partition directory
     # Note : Partitions have to be created and configured before running slapgrid
@@ -351,15 +360,14 @@ class Partition(object):
     self.updateSymlink(sr_symlink, self.software_path)
 
     instance_stat_info = os.stat(self.instance_path)
-    permission = oct(stat.S_IMODE(instance_stat_info.st_mode))
+    permission = stat.S_IMODE(instance_stat_info.st_mode)
     if permission != REQUIRED_COMPUTER_PARTITION_PERMISSION:
-      raise WrongPermissionError('Wrong permissions in %s : actual ' \
-                                          'permissions are : %s, wanted ' \
-                                          'are %s' %
+      raise WrongPermissionError('Wrong permissions in %s: actual '
+                                 'permissions are: 0%o, wanted are 0%o' %
                                  (self.instance_path, permission,
                                   REQUIRED_COMPUTER_PARTITION_PERMISSION))
-    os.environ = utils.getCleanEnvironment(pwd.getpwuid(
-      instance_stat_info.st_uid).pw_dir)
+    os.environ = getCleanEnvironment(logger=self.logger,
+                                     home_path=pwd.getpwuid(instance_stat_info.st_uid).pw_dir)
     # Generates buildout part from template
     template_location = os.path.join(self.software_path, 'instance.cfg')
     # Backward compatibility: "instance.cfg" file was named "template.cfg".
@@ -369,23 +377,23 @@ class Partition(object):
     self.logger.debug("Copying %r to %r" % (template_location, config_location))
     try:
       shutil.copy(template_location, config_location)
-    except IOError, e:
+    except IOError as exc:
       # Template not found on SR, we notify user.
-      raise IOError('Software Release %s is not correctly installed.\n'
-          '%s' % (self.software_release_url, e))
+      raise IOError('Software Release %s is not correctly installed.\n%s' % (
+                      self.software_release_url, exc))
     # fill generated buildout with additional information
     buildout_text = open(config_location).read()
     buildout_text += '\n\n' + pkg_resources.resource_string(__name__,
-        'templates/buildout-tail.cfg.in') % dict(
-      computer_id=self.computer_id,
-      partition_id=self.partition_id,
-      server_url=self.server_url,
-      software_release_url=self.software_release_url,
-      key_file=self.key_file,
-      cert_file=self.cert_file
-    )
+        'templates/buildout-tail.cfg.in') % {
+                'computer_id': self.computer_id,
+                'partition_id': self.partition_id,
+                'server_url': self.server_url,
+                'software_release_url': self.software_release_url,
+                'key_file': self.key_file,
+                'cert_file': self.cert_file,
+                }
     open(config_location, 'w').write(buildout_text)
-    os.chmod(config_location, 0640)
+    os.chmod(config_location, 0o640)
     # Try to find the best possible buildout:
     #  *) if software_root/bin/bootstrap exists use this one to bootstrap
     #     locally
@@ -418,12 +426,14 @@ class Partition(object):
 
       self.logger.debug('Invoking %r in %r' % (' '.join(invocation_list),
         self.instance_path))
-      process_handler = utils.SlapPopen(invocation_list,
-                                        preexec_fn=lambda: utils.dropPrivileges(uid, gid),
+      process_handler = SlapPopen(invocation_list,
+                                  preexec_fn=lambda: dropPrivileges(uid, gid, logger=self.logger),
                                   cwd=self.instance_path,
-                                        env=utils.getCleanEnvironment(pwd.getpwuid(uid).pw_dir),
+                                  env=getCleanEnvironment(logger=self.logger,
+                                                          home_path=pwd.getpwuid(uid).pw_dir),
                                   stdout=subprocess.PIPE,
-                                        stderr=subprocess.STDOUT)
+                                  stderr=subprocess.STDOUT,
+                                  logger=self.logger)
       if process_handler.returncode is None or process_handler.returncode != 0:
         message = 'Failed to bootstrap buildout in %r.' % (self.instance_path)
         self.logger.error(message)
@@ -432,18 +442,23 @@ class Partition(object):
 
     if not os.path.exists(buildout_binary):
       # use own buildout generation
-      utils.bootstrapBuildout(self.instance_path, self.buildout,
-        ['buildout:bin-directory=%s'% os.path.join(self.instance_path,
-        'sbin')])
+      utils.bootstrapBuildout(path=self.instance_path,
+                              buildout=self.buildout,
+                              logger=self.logger,
+                              additional_buildout_parameter_list=
+                                ['buildout:bin-directory=%s' %
+                                    os.path.join(self.instance_path, 'sbin')])
       buildout_binary = os.path.join(self.instance_path, 'sbin', 'buildout')
     # Launches buildout
-    utils.launchBuildout(self.instance_path, buildout_binary)
+    utils.launchBuildout(path=self.instance_path,
+                         buildout_binary=buildout_binary,
+                         logger=self.logger)
     # Generates supervisord configuration file from template
     self.logger.info("Generating supervisord config file from template...")
     # check if CP/etc/run exists and it is a directory
     # iterate over each file in CP/etc/run
     # iterate over each file in CP/etc/service adding WatchdogID to their name
-    # if at least one is not 0750 raise -- partition has something funny
+    # if at least one is not 0o750 raise -- partition has something funny
     runner_list = []
     service_list = []
     if os.path.exists(self.run_path):
@@ -461,15 +476,16 @@ class Partition(object):
       partition_id = self.computer_partition.getId()
       group_partition_template = pkg_resources.resource_stream(__name__,
           'templates/group_partition_supervisord.conf.in').read()
-      self.partition_supervisor_configuration = group_partition_template % dict(
-          instance_id=partition_id,
-          program_list=','.join(['_'.join([partition_id, runner])
-            for runner in runner_list+service_list]))
+      self.partition_supervisor_configuration = group_partition_template % {
+              'instance_id': partition_id,
+              'program_list': ','.join(['_'.join([partition_id, runner])
+                                        for runner in runner_list + service_list])
+              }
       # Same method to add to service and run
-      self.addServiceToGroup(partition_id, runner_list,self.run_path)
-      self.addServiceToGroup(partition_id, service_list,self.service_path,
+      self.addServiceToGroup(partition_id, runner_list, self.run_path)
+      self.addServiceToGroup(partition_id, service_list, self.service_path,
                              extension=getWatchdogID())
-      utils.updateFile(self.supervisord_partition_configuration_path,
+      updateFile(self.supervisord_partition_configuration_path,
                  self.partition_supervisor_configuration)
     self.updateSupervisor()
 
@@ -481,21 +497,21 @@ class Partition(object):
     partition_id = self.computer_partition.getId()
     try:
       supervisor.startProcessGroup(partition_id, False)
-    except xmlrpclib.Fault, e:
-      if e.faultString.startswith('BAD_NAME:'):
-        self.logger.info("Nothing to start on %s..." % \
+    except xmlrpclib.Fault as exc:
+      if exc.faultString.startswith('BAD_NAME:'):
+        self.logger.info("Nothing to start on %s..." %
                          self.computer_partition.getId())
     else:
       self.logger.info("Requested start of %s..." % self.computer_partition.getId())
 
   def stop(self):
     """Asks supervisord to stop the instance."""
-    supervisor = self.getSupervisorRPC()
     partition_id = self.computer_partition.getId()
     try:
+      supervisor = self.getSupervisorRPC()
       supervisor.stopProcessGroup(partition_id, False)
-    except xmlrpclib.Fault, e:
-      if e.faultString.startswith('BAD_NAME:'):
+    except xmlrpclib.Fault as exc:
+      if exc.faultString.startswith('BAD_NAME:'):
         self.logger.info('Partition %s not known in supervisord, ignoring' % partition_id)
     else:
       self.logger.info("Requested stop of %s..." % self.computer_partition.getId())
@@ -503,7 +519,7 @@ class Partition(object):
   def destroy(self):
     """Destroys the partition and makes it available for subsequent use."
     """
-    self.logger.info("Destroying Computer Partition %s..." \
+    self.logger.info("Destroying Computer Partition %s..."
         % self.computer_partition.getId())
     # Launches "destroy" binary if exists
     destroy_executable_location = os.path.join(self.instance_path, 'sbin',
@@ -511,12 +527,14 @@ class Partition(object):
     if os.path.exists(destroy_executable_location):
       uid, gid = self.getUserGroupId()
       self.logger.debug('Invoking %r' % destroy_executable_location)
-      process_handler = utils.SlapPopen([destroy_executable_location],
-                                        preexec_fn=lambda: utils.dropPrivileges(uid, gid),
+      process_handler = SlapPopen([destroy_executable_location],
+                                  preexec_fn=lambda: dropPrivileges(uid, gid, logger=self.logger),
                                   cwd=self.instance_path,
-                                        env=utils.getCleanEnvironment(pwd.getpwuid(uid).pw_dir),
+                                  env=getCleanEnvironment(logger=self.logger,
+                                                          home_path=pwd.getpwuid(uid).pw_dir),
                                   stdout=subprocess.PIPE,
-                                        stderr=subprocess.STDOUT)
+                                  stderr=subprocess.STDOUT,
+                                  logger=self.logger)
       if process_handler.returncode is None or process_handler.returncode != 0:
         message = 'Failed to destroy Computer Partition in %r.' % \
             self.instance_path
@@ -542,10 +560,8 @@ class Partition(object):
         if os.path.exists(self.supervisord_partition_configuration_path):
           os.remove(self.supervisord_partition_configuration_path)
         self.updateSupervisor()
-    except IOError as error:
-      error_string = "I/O error while freeing partition (%s): %s" \
-                     % (self.instance_path, error)
-      raise IOError(error_string)
+    except IOError as exc:
+      raise IOError("I/O error while freeing partition (%s): %s" % (self.instance_path, exc))
 
   def fetchInformations(self):
     """Fetch usage informations with buildout, returns it.
@@ -589,4 +605,3 @@ class Partition(object):
       supervisor.addProcessGroup(gname)
       self.logger.info('Updated %r' % gname)
     self.logger.debug('Supervisord updated')
-

slapos/grid/slapgrid.py

@@ -28,12 +28,6 @@
 #
 ##############################################################################
 
-import argparse
-import ConfigParser
-from exception import BuildoutFailedError
-from hashlib import md5
-from lxml import etree
-import logging
 import os
 import pkg_resources
 import random
@@ -45,30 +39,22 @@ import tempfile
 import time
 import traceback
 import warnings
+
 if sys.version_info < (2, 6):
-  warnings.warn('Used python version (%s) is old and have problems with'
+  warnings.warn('Used python version (%s) is old and has problems with'
       ' IPv6 connections' % sys.version.split('\n')[0])
 
+from lxml import etree
+
 from slapos.slap.slap import NotFoundError
 from slapos.slap.slap import ServerError
-from SlapObject import Software, Partition, WrongPermissionError, \
-    PathDoesNotExistError
-from svcbackend import launchSupervisord
-from utils import createPrivateDirectory
-from utils import dropPrivileges
-from utils import getSoftwareUrlHash
-from utils import setRunning
-from utils import setFinished
-from utils import SlapPopen
-from utils import updateFile
-from slapos import slap
-
-MANDATORY_PARAMETER_LIST = [
-    'computer_id',
-    'instance_root',
-    'master_url',
-    'software_root',
-]
+from slapos.grid.exception import BuildoutFailedError
+from slapos.grid.SlapObject import Software, Partition
+from slapos.grid.svcbackend import launchSupervisord
+from slapos.grid.utils import (md5digest, createPrivateDirectory, dropPrivileges,
+                               SlapPopen, updateFile)
+import slapos.slap
+
 
 # XXX: should be moved to SLAP library
 COMPUTER_PARTITION_DESTROYED_STATE = 'destroyed'
@@ -90,310 +76,156 @@ class _formatXMLError(Exception):
   pass
 
 
-def parseArgumentTupleAndReturnSlapgridObject(*argument_tuple):
-  """Parses arguments either from command line, from method parameters or from
-     config file. Then returns a new instance of slapgrid.Slapgrid with those
-     parameters. Also returns the options dict and unused variable list, and
-     configures logger.
-  """
-  parser = argparse.ArgumentParser()
-  parser.add_argument("--instance-root",
-      help="The instance root directory location.")
-  parser.add_argument("--software-root",
-      help="The software_root directory location.")
-  parser.add_argument("--master-url",
-      help="The master server URL. Mandatory.")
-  parser.add_argument("--computer-id",
-      help="The computer id defined in the server.")
-  parser.add_argument("--supervisord-socket",
-      help="The socket supervisor will use.")
-  parser.add_argument("--supervisord-configuration-path",
-      help="The location where supervisord configuration will be stored.")
-  parser.add_argument("--buildout", default=None,
-      help="Location of buildout binary.")
-  parser.add_argument("--pidfile",
-      help="The location where pidfile will be created.")
-  parser.add_argument("--logfile",
-      help="The location where slapgrid logfile will be created.")
-  parser.add_argument("--key_file", help="SSL Authorisation key file.")
-  parser.add_argument("--cert_file",
-      help="SSL Authorisation certificate file.")
-  parser.add_argument("--signature_private_key_file",
-      help="Signature private key file.")
-  parser.add_argument("--master_ca_file",
-      help="Root certificate of SlapOS master key.")
-  parser.add_argument("--certificate_repository_path",
-      help="Path to directory where downloaded certificates would be stored.")
-  parser.add_argument("-c", "--console", action="store_true", default=False,
-      help="Deprecated, doesn't do anything.")
-  parser.add_argument("-v", "--verbose", action="store_true", default=False,
-      help="Be verbose.")
-  parser.add_argument("--maximum-periodicity", type=int, default=None,
-      help="Periodicity at which buildout should be run in instance.")
-  parser.add_argument("--promise-timeout", type=int, default=3,
-      help="Promise timeout in seconds.")
-  parser.add_argument("--now", action="store_true", default=False,
-      help="Launch slapgrid without delay. Default behavior.")
-  parser.add_argument("--all", action="store_true", default=False,
-      help="Launch slapgrid to process all Softare Releases "
-           "and/or Computer Partitions.")
-  parser.add_argument("--only-sr",
-      help="Force the update of a single software release (use url hash), "
-           "even if is already installed. This option will make all others "
-           "sofware releases be ignored.")
-  parser.add_argument("--only-cp",
-      help="Update a single or a list of computer partitions "
-           "(ie.:slappartX, slappartY),"
-           "this option will make all others computer partitions be ignored.")
-
-  parser.add_argument("configuration_file", nargs=1, type=argparse.FileType(),
-      help="SlapOS configuration file.")
-
-  # Deprecated options
-  parser.add_argument("--develop", action="store_true", default=False,
-      help="Deprecated, same as --all.")
-  parser.add_argument("--only_sr",
-      help="Deprecated, same as --only-sr.")
-  parser.add_argument("--only_cp",
-      help="Deprecated, same as --only-cp.")
-  parser.add_argument("--maximal_delay",
-      help="Deprecated. Will only work from configuration file in the future.")
+def check_missing_parameters(options):
+  required = set([
+      'computer_id',
+      'instance_root',
+      'master_url',
+      'software_root',
+  ])
 
+  if 'key_file' in options:
+    required.add('certificate_repository_path')
+    required.add('cert_file')
+  if 'cert_file' in options:
+    required.add('certificate_repository_path')
+    required.add('key_file')
+
+  missing = required.difference(options)
+
+  if missing:
+    raise RuntimeError('Missing mandatory parameters: %s' % ', '.join(sorted(missing)))
 
-  # Parses arguments
-  if argument_tuple == ():
-    # No arguments given to entry point : we parse sys.argv.
-    argument_option_instance = parser.parse_args()
-  else:
-    argument_option_instance = \
-      parser.parse_args(list(argument_tuple))
-  # Parses arguments from config file, if needed, then merge previous arguments
-  option_dict = {}
-  configuration_file = argument_option_instance.configuration_file[0]
-  # Loads config (if config specified)
-  slapgrid_configuration = ConfigParser.SafeConfigParser()
-  slapgrid_configuration.readfp(configuration_file)
-  # Merges the two dictionnaries
-  option_dict = dict(slapgrid_configuration.items("slapos"))
-  if slapgrid_configuration.has_section("networkcache"):
-    option_dict.update(dict(slapgrid_configuration.items("networkcache")))
-  for argument_key, argument_value in vars(argument_option_instance
-      ).iteritems():
-    if argument_value is not None:
-      option_dict.update({argument_key: argument_value})
-  # Configures logger.
-  if option_dict['verbose']:
-    level = logging.DEBUG
-  else:
-    level = logging.INFO
-  logging.basicConfig(format='%(asctime)s %(levelname)-8s %(message)s',
-                      level=level,
-                      datefmt='%Y-%m-%dT%H:%M:%S')
-  if option_dict.get('logfile'):
-    console = logging.FileHandler(option_dict['logfile'])
-    console.setLevel(level)
-    console.setFormatter(logging.Formatter(
-        '%(asctime)s %(name)-18s: %(levelname)-8s %(message)s'))
-    logging.getLogger('').addHandler(console)
-
-  missing_mandatory_parameter_list = []
-  for mandatory_parameter in MANDATORY_PARAMETER_LIST:
-    if not mandatory_parameter in option_dict:
-      missing_mandatory_parameter_list.append(mandatory_parameter)
-
-  if option_dict.get('all') is True:
-    option_dict['develop'] = True
-
-  if option_dict.get('maximum_periodicity') is not None:
-    option_dict['force_periodicity'] = True
-
-  repository_required = False
-  if 'key_file' in option_dict:
-    repository_required = True
-    if not 'cert_file' in option_dict:
-      missing_mandatory_parameter_list.append('cert_file')
-  if 'cert_file' in option_dict:
-    repository_required = True
-    if not 'key_file' in option_dict:
-      missing_mandatory_parameter_list.append('key_file')
-  if repository_required:
-    if 'certificate_repository_path' not in option_dict:
-      missing_mandatory_parameter_list.append('certificate_repository_path')
-
-  if len(missing_mandatory_parameter_list) > 0:
-    parser.error('Missing mandatory parameters:\n%s' % '\n'.join(
-      missing_mandatory_parameter_list))
-
-  key_file = option_dict.get('key_file')
-  cert_file = option_dict.get('cert_file')
-  master_ca_file = option_dict.get('master_ca_file')
-  signature_private_key_file = option_dict.get('signature_private_key_file')
-
-  mandatory_file_list = [key_file, cert_file, master_ca_file]
-  # signature_private_key_file is not mandatory, we must be able to run
-  # slapgrid scripts without this parameter.
-  if signature_private_key_file:
-    mandatory_file_list.append(signature_private_key_file)
-
-  for k in ['shacache-cert-file', 'shacache-key-file', 'shadir-cert-file',
-      'shadir-key-file']:
-    mandatory_file_list.append(option_dict.get(k, None))
-
-  for f in mandatory_file_list:
-    if f is not None:
-      if not os.path.exists(f):
-        parser.error('File %r does not exist.' % f)
-
-  certificate_repository_path = option_dict.get('certificate_repository_path')
-  if certificate_repository_path is not None:
-    if not os.path.isdir(certificate_repository_path):
-      parser.error('Directory %r does not exist' %
-          certificate_repository_path)
+
+def check_missing_files(options):
+  req_files = [
+          options.get('key_file'),
+          options.get('cert_file'),
+          options.get('master_ca_file'),
+          options.get('shacache-cert-file'),
+          options.get('shacache-key-file'),
+          options.get('shadir-cert-file'),
+          options.get('shadir-key-file'),
+          options.get('signature_private_key_file')
+          ]
+
+  req_dirs = [
+          options.get('certificate_repository_path')
+          ]
+
+  for f in req_files:
+    if f and not os.path.exists(f):
+        raise RuntimeError('File %r does not exist.' % f)
+
+  for d in req_dirs:
+    if d and not os.path.isdir(d):
+      raise RuntimeError('Directory %r does not exist' % d)
+
+
+def merged_options(args, configp):
+  options = dict(configp.items('slapos'))
+
+  if configp.has_section('networkcache'):
+    options.update(dict(configp.items('networkcache')))
+  for key, value in vars(args).iteritems():
+    if value is not None:
+      options[key] = value
+
+  if options.get('all'):
+    options['develop'] = True
+
+  if options.get('maximum_periodicity') is not None:
+    options['force_periodicity'] = True
 
   # Supervisord configuration location
-  if not option_dict.get('supervisord_configuration_path'):
-    option_dict['supervisord_configuration_path'] = \
-      os.path.join(option_dict['instance_root'], 'etc', 'supervisord.conf')
+  if not options.get('supervisord_configuration_path'):
+    options['supervisord_configuration_path'] = \
+      os.path.join(options['instance_root'], 'etc', 'supervisord.conf')
   # Supervisord socket
-  if not option_dict.get('supervisord_socket'):
-    option_dict['supervisord_socket'] = \
-      os.path.join(option_dict['instance_root'], 'supervisord.socket')
-
-  signature_certificate_list_string = \
-    option_dict.get('signature-certificate-list', None)
-  if signature_certificate_list_string is not None:
-    cert_marker = "-----BEGIN CERTIFICATE-----"
-    signature_certificate_list = [cert_marker + '\n' + q.strip() \
-      for q in signature_certificate_list_string.split(cert_marker) \
-        if q.strip()]
-  else:
-    signature_certificate_list = None
+  if not options.get('supervisord_socket'):
+    options['supervisord_socket'] = \
+      os.path.join(options['instance_root'], 'supervisord.socket')
 
   # Parse cache / binary cache options
   # Backward compatibility about "binary-cache-url-blacklist" deprecated option
-  if option_dict.get("binary-cache-url-blacklist") and not \
-      option_dict.get("download-from-binary-cache-url-blacklist"):
-    option_dict["download-from-binary-cache-url-blacklist"] = \
-        option_dict["binary-cache-url-blacklist"]
-  option_dict["download-from-binary-cache-url-blacklist"] = [
-      url.strip() for url in option_dict.get(
+  if options.get("binary-cache-url-blacklist") and not \
+      options.get("download-from-binary-cache-url-blacklist"):
+    options["download-from-binary-cache-url-blacklist"] = \
+        options["binary-cache-url-blacklist"]
+  options["download-from-binary-cache-url-blacklist"] = [
+      url.strip() for url in options.get(
           "download-from-binary-cache-url-blacklist", "").split('\n') if url]
-  option_dict["upload-to-binary-cache-url-blacklist"] = [
-      url.strip() for url in option_dict.get(
+  options["upload-to-binary-cache-url-blacklist"] = [
+      url.strip() for url in options.get(
           "upload-to-binary-cache-url-blacklist", "").split('\n') if url]
 
-  # Sleep for a random time to avoid SlapOS Master being DDOSed by an army of
-  # SlapOS Nodes configured with cron.
-  if option_dict["now"]:
-    # XXX-Cedric: deprecate "--now"
-    maximal_delay = 0
-  else:
-    maximal_delay = int(option_dict.get("maximal_delay", "0"))
-  if maximal_delay > 0:
+  return options
+
+
+def random_delay(options, logger):
+  """
+  Sleep for a random time to avoid SlapOS Master being DDOSed by an army of
+  SlapOS Nodes configured with cron.
+  """
+  if options['now']:
+    # XXX-Cedric: deprecate '--now'
+    return
+
+  maximal_delay = int(options.get('maximal_delay', '0'))
+  if maximal_delay:
     duration = random.randint(1, maximal_delay)
-    logging.info("Sleeping for %s seconds. To disable this feature, " \
-                    "check --now parameter in slapgrid help." % duration)
+    logger.info('Sleeping for %s seconds. To disable this feature, ' \
+                'check --now parameter in slapgrid help.' % duration)
     time.sleep(duration)
 
-  # Return new Slapgrid instance and options
-  return ([Slapgrid(software_root=option_dict['software_root'],
-            instance_root=option_dict['instance_root'],
-            master_url=option_dict['master_url'],
-            computer_id=option_dict['computer_id'],
-            supervisord_socket=option_dict['supervisord_socket'],
-            supervisord_configuration_path=option_dict[
-              'supervisord_configuration_path'],
-            key_file=key_file,
-            cert_file=cert_file,
-            master_ca_file=master_ca_file,
-            certificate_repository_path=certificate_repository_path,
-            signature_private_key_file=signature_private_key_file,
+
+def create_slapgrid_object(options, logger):
+  signature_certificate_list = None
+  if 'signature-certificate-list' in options:
+    cert_marker = '-----BEGIN CERTIFICATE-----'
+    signature_certificate_list = [
+            cert_marker + '\n' + q.strip()
+            for q in options['signature-certificate-list'].split(cert_marker)
+            if q.strip()
+            ]
+
+  op = options
+  return Slapgrid(software_root=op['software_root'],
+                  instance_root=op['instance_root'],
+                  master_url=op['master_url'],
+                  computer_id=op['computer_id'],
+                  supervisord_socket=op['supervisord_socket'],
+                  supervisord_configuration_path=op['supervisord_configuration_path'],
+                  buildout=op.get('buildout'),
+                  logger=logger,
+                  force_periodicity = op.get('force_periodicity', False),
+                  maximum_periodicity = op.get('maximum_periodicity', 86400),
+                  key_file=op.get('key_file'),
+                  cert_file=op.get('cert_file'),
+                  signature_private_key_file=op.get('signature_private_key_file'),
                   signature_certificate_list=signature_certificate_list,
-            download_binary_cache_url=\
-              option_dict.get('download-binary-cache-url', None),
-            upload_binary_cache_url=\
-              option_dict.get('upload-binary-cache-url', None),
+                  download_binary_cache_url=op.get('download-binary-cache-url'),
+                  upload_binary_cache_url=op.get('upload-binary-cache-url'),
                   download_from_binary_cache_url_blacklist=\
-                option_dict.get('download-from-binary-cache-url-blacklist', []),
+                      op.get('download-from-binary-cache-url-blacklist', []),
                   upload_to_binary_cache_url_blacklist=\
-                option_dict.get('upload-to-binary-cache-url-blacklist', []),
-            upload_cache_url=option_dict.get('upload-cache-url', None),
-            download_binary_dir_url=\
-              option_dict.get('download-binary-dir-url', None),
-            upload_binary_dir_url=\
-              option_dict.get('upload-binary-dir-url', None),
-            upload_dir_url=option_dict.get('upload-dir-url', None),
-            buildout=option_dict.get('buildout'),
-            promise_timeout=option_dict['promise_timeout'],
-            shacache_cert_file=option_dict.get('shacache-cert-file', None),
-            shacache_key_file=option_dict.get('shacache-key-file', None),
-            shadir_cert_file=option_dict.get('shadir-cert-file', None),
-            shadir_key_file=option_dict.get('shadir-key-file', None),
-            develop=option_dict.get('develop', False),
-            software_release_filter_list=option_dict.get('only-sr',
+                      op.get('upload-to-binary-cache-url-blacklist', []),
+                  upload_cache_url=op.get('upload-cache-url'),
+                  download_binary_dir_url=op.get('download-binary-dir-url'),
+                  upload_binary_dir_url=op.get('upload-binary-dir-url'),
+                  upload_dir_url=op.get('upload-dir-url'),
+                  master_ca_file=op.get('master_ca_file'),
+                  certificate_repository_path=op.get('certificate_repository_path'),
+                  promise_timeout=op['promise_timeout'],
+                  shacache_cert_file=op.get('shacache-cert-file'),
+                  shacache_key_file=op.get('shacache-key-file'),
+                  shadir_cert_file=op.get('shadir-cert-file'),
+                  shadir_key_file=op.get('shadir-key-file'),
+                  develop=op.get('develop', False),
                   # Try to fetch from deprecated argument
-                option_dict.get('only_sr', None)),
-            computer_partition_filter_list=option_dict.get('only-cp',
+                  software_release_filter_list=op.get('only-sr', op.get('only_sr')),
                   # Try to fetch from deprecated argument
-                option_dict.get('only_cp', None)),
-            force_periodicity = option_dict.get('force_periodicity', False),
-            maximum_periodicity = option_dict.get('maximum_periodicity', 86400),
-            ),
-          option_dict])
-
-
-def realRun(argument_tuple, method_list):
-  slapgrid_object, option_dict = \
-      parseArgumentTupleAndReturnSlapgridObject(*argument_tuple)
-  pidfile = option_dict.get('pidfile')
-  if pidfile:
-    setRunning(pidfile)
-  try:
-    failed = False
-    failed_promise = False
-    for method in method_list:
-      # Quite complicated way to figure out if everything went fine
-      return_value = getattr(slapgrid_object, method)()
-      if return_value == SLAPGRID_FAIL:
-        failed = True
-      if return_value == SLAPGRID_PROMISE_FAIL:
-        failed_promise = True
-  finally:
-    if pidfile:
-      setFinished(pidfile)
-  if failed:
-    sys.exit(SLAPGRID_FAIL)
-  if failed_promise:
-    sys.exit(SLAPGRID_PROMISE_FAIL)
-  sys.exit(SLAPGRID_SUCCESS)
-
-
-def run(*argument_tuple):
-  """Hooks for generic entry point to proces Software Releases (sr),
-     Computer Partitions (cp) and Usage Reports (ur)
-     Will run one by one each task (sr, cp, ur). If specified,
-     will run in the user wanted order.
-  """
-  realRun(argument_tuple, ['processSoftwareReleaseList',
-    'processComputerPartitionList', 'agregateAndSendUsage'])
-
-
-def runSoftwareRelease(*argument_tuple):
-  """Hook for entry point to process Software Releases only
-  """
-  realRun(argument_tuple, ['processSoftwareReleaseList'])
-
-
-def runComputerPartition(*argument_tuple):
-  """Hook for entry point to process Computer Partitions only
-  """
-  realRun(argument_tuple, ['processComputerPartitionList'])
-
-
-def runUsageReport(*argument_tuple):
-  """Hook for entry point to process Usage Reports only
-  """
-  realRun(argument_tuple, ['agregateAndSendUsage'])
+                  computer_partition_filter_list=op.get('only-cp', op.get('only_cp')))
 
 
 class Slapgrid(object):
@@ -412,6 +244,7 @@ class Slapgrid(object):
                supervisord_socket,
                supervisord_configuration_path,
                buildout,
+               logger,
                force_periodicity=False,
                maximum_periodicity=86400,
                key_file=None,
@@ -465,17 +298,15 @@ class Slapgrid(object):
     self.shacache_key_file = shacache_key_file
     self.shadir_cert_file = shadir_cert_file
     self.shadir_key_file = shadir_key_file
-    # Configures logger
-    self.logger = logging.getLogger('Slapgrid')
+    self.logger = logger
     # Creates objects from slap module
-    self.slap = slap.slap()
+    self.slap = slapos.slap.slap()
     self.slap.initializeConnection(self.master_url, key_file=self.key_file,
         cert_file=self.cert_file, master_ca_file=self.master_ca_file)
     self.computer = self.slap.registerComputer(self.computer_id)
     # Defines all needed paths
-    self.instance_etc_directory = os.path.join(self.instance_root, 'etc')
     self.supervisord_configuration_directory = \
-        os.path.join(self.instance_etc_directory, 'supervisord.conf.d')
+        os.path.join(self.instance_root, 'etc', 'supervisord.conf.d')
     self.buildout = buildout
     self.promise_timeout = promise_timeout
     self.develop = develop
@@ -494,7 +325,7 @@ class Slapgrid(object):
   def getWatchdogLine(self):
     invocation_list = [WATCHDOG_PATH]
     invocation_list.append("--master-url '%s' " % self.master_url)
-    if self.certificate_repository_path is not None:
+    if self.certificate_repository_path:
       invocation_list.append("--certificate-repository-path '%s'" \
                                % self.certificate_repository_path)
     invocation_list.append("--computer-id '%s'" % self.computer_id)
@@ -506,64 +337,58 @@ class Slapgrid(object):
     """
     # Checks for software_root and instance_root existence
     if not os.path.isdir(self.software_root):
-      error = "%s does not exist." % self.software_root
-      raise OSError(error)
+      raise OSError('%s does not exist.' % self.software_root)
     if not os.path.isdir(self.instance_root):
-      error = "%s does not exist." % self.instance_root
-      raise OSError(error)
+      raise OSError('%s does not exist.' % self.instance_root)
     # Creates everything needed
-    try:
+
     # Creates instance_root structure
-      createPrivateDirectory(self.instance_etc_directory)
     createPrivateDirectory(os.path.join(self.instance_root, 'var'))
     createPrivateDirectory(os.path.join(self.instance_root, 'var', 'log'))
     createPrivateDirectory(os.path.join(self.instance_root, 'var', 'run'))
+
+    createPrivateDirectory(os.path.join(self.instance_root, 'etc'))
     createPrivateDirectory(self.supervisord_configuration_directory)
+
     # Creates supervisord configuration
     updateFile(self.supervisord_configuration_path,
       pkg_resources.resource_stream(__name__,
-          'templates/supervisord.conf.in').read() % dict(
-            supervisord_configuration_directory=\
-                self.supervisord_configuration_directory,
-            supervisord_socket=os.path.abspath(self.supervisord_socket),
-            supervisord_loglevel='info',
-            supervisord_logfile=os.path.abspath(os.path.join(
-              self.instance_root, 'var', 'log', 'supervisord.log')),
-            supervisord_logfile_maxbytes='50MB',
-            supervisord_nodaemon='false',
-            supervisord_pidfile=os.path.abspath(os.path.join(
-              self.instance_root, 'var', 'run', 'supervisord.pid')),
-            supervisord_logfile_backups='10',
-            watchdog_command = self.getWatchdogLine(),
-          ))
-    except (WrongPermissionError, PathDoesNotExistError) as error:
-      raise error
+        'templates/supervisord.conf.in').read() % {
+            'supervisord_configuration_directory': self.supervisord_configuration_directory,
+            'supervisord_socket': os.path.abspath(self.supervisord_socket),
+            'supervisord_loglevel': 'info',
+            'supervisord_logfile': os.path.abspath(os.path.join(self.instance_root, 'var', 'log', 'supervisord.log')),
+            'supervisord_logfile_maxbytes': '50MB',
+            'supervisord_nodaemon': 'false',
+            'supervisord_pidfile': os.path.abspath(os.path.join(self.instance_root, 'var', 'run', 'supervisord.pid')),
+            'supervisord_logfile_backups': '10',
+            'watchdog_command': self.getWatchdogLine(),
+            })
 
   def getComputerPartitionList(self):
     try:
-      computer_partition_list = self.computer.getComputerPartitionList()
-    except socket.error as error:
-      self.logger.fatal(error)
+      return self.computer.getComputerPartitionList()
+    except socket.error as exc:
+      self.logger.fatal(exc)
       raise
-    return computer_partition_list
 
   def processSoftwareReleaseList(self):
     """Will process each Software Release.
     """
     self.checkEnvironmentAndCreateStructure()
-    logger = logging.getLogger('SoftwareReleases')
-    logger.info("Processing software releases...")
+    self.logger.info('Processing software releases...')
     # Boolean to know if every instance has correctly been deployed
     clean_run = True
     for software_release in self.computer.getSoftwareReleaseList():
       state = software_release.getState()
       try:
         software_release_uri = software_release.getURI()
-        url_hash = md5(software_release_uri).hexdigest()
+        url_hash = md5digest(software_release_uri)
         software_path = os.path.join(self.software_root, url_hash)
         software = Software(url=software_release_uri,
             software_root=self.software_root,
             buildout=self.buildout,
+            logger=self.logger,
             signature_private_key_file=self.signature_private_key_file,
             signature_certificate_list=self.signature_certificate_list,
             download_binary_cache_url=self.download_binary_cache_url,
@@ -585,43 +410,40 @@ class Slapgrid(object):
           if self.develop or (not os.path.exists(completed_tag) and \
                  len(self.software_release_filter_list) == 0) or \
                  url_hash in self.software_release_filter_list or \
-                 url_hash in (md5(uri).hexdigest() for uri in self.software_release_filter_list):
+                 url_hash in (md5digest(uri) for uri in self.software_release_filter_list):
             try:
               software_release.building()
             except NotFoundError:
               pass
             software.install()
-            file_descriptor = open(completed_tag, 'w')
-            file_descriptor.write(time.asctime())
-            file_descriptor.close()
+            with open(completed_tag, 'w') as fout:
+              fout.write(time.asctime())
         elif state == 'destroyed':
           if os.path.exists(software_path):
-            logger.info('Destroying %r...' % software_release_uri)
+            self.logger.info('Destroying %r...' % software_release_uri)
             software.destroy()
-            logger.info('Destroyed %r.' % software_release_uri)
+            self.logger.info('Destroyed %r.' % software_release_uri)
       # Send log before exiting
       except (SystemExit, KeyboardInterrupt):
-        exception = traceback.format_exc()
-        software_release.error(exception)
+        software_release.error(traceback.format_exc(), logger=self.logger)
         raise
 
       # Buildout failed: send log but don't print it to output (already done)
-      except BuildoutFailedError as exception:
+      except BuildoutFailedError as exc:
         clean_run = False
         try:
-          software_release.error(exception)
+          software_release.error(exc, logger=self.logger)
         except (SystemExit, KeyboardInterrupt):
           raise
         except Exception:
-          exception = traceback.format_exc()
-          logger.error('Problem during reporting error, continuing:\n' +
-            exception)
+          self.logger.error('Problem while reporting error, continuing:\n%s' %
+                            traceback.format_exc())
 
       # For everything else: log it, send it, continue.
       except Exception:
-        exception = traceback.format_exc()
-        logger.error(exception)
-        software_release.error(exception)
+        exc = traceback.format_exc()
+        self.logger.error(exc)
+        software_release.error(exc, logger=self.logger)
         clean_run = False
       else:
         if state == 'available':
@@ -634,7 +456,7 @@ class Slapgrid(object):
             software_release.destroyed()
           except (NotFoundError, ServerError):
             print traceback.format_exc()
-    logger.info("Finished software releases.")
+    self.logger.info('Finished software releases.')
 
     # Return success value
     if not clean_run:
@@ -644,12 +466,12 @@ class Slapgrid(object):
 
   def _launchSupervisord(self):
     launchSupervisord(self.supervisord_socket,
-        self.supervisord_configuration_path)
+                      self.supervisord_configuration_path,
+                      logger=self.logger)
 
   def _checkPromises(self, computer_partition):
     self.logger.info("Checking promises...")
-    instance_path = os.path.join(self.instance_root,
-        computer_partition.getId())
+    instance_path = os.path.join(self.instance_root, computer_partition.getId())
 
     uid, gid = None, None
     stat_info = os.stat(instance_path)
@@ -662,11 +484,8 @@ class Slapgrid(object):
     # Get the list of promises
     promise_dir = os.path.join(instance_path, 'etc', 'promise')
     if os.path.exists(promise_dir) and os.path.isdir(promise_dir):
-      cwd = instance_path
-      promises_list = os.listdir(promise_dir)
-
       # Check whether every promise is kept
-      for promise in promises_list:
+      for promise in os.listdir(promise_dir):
         promise_present = True
 
         command = [os.path.join(promise_dir, promise)]
@@ -674,13 +493,13 @@ class Slapgrid(object):
         promise = os.path.basename(command[0])
         self.logger.info("Checking promise %r.", promise)
 
-        kw = dict(stdout=subprocess.PIPE, stderr=subprocess.PIPE,
-            stdin=subprocess.PIPE)
-
         process_handler = subprocess.Popen(command,
-          preexec_fn=lambda: dropPrivileges(uid, gid),
-          cwd=cwd,
-          env=None if sys.platform == 'cygwin' else {}, **kw)
+                                           preexec_fn=lambda: dropPrivileges(uid, gid, logger=self.logger),
+                                           cwd=instance_path,
+                                           env=None if sys.platform == 'cygwin' else {},
+                                           stdout=subprocess.PIPE,
+                                           stderr=subprocess.PIPE,
+                                           stdin=subprocess.PIPE)
         process_handler.stdin.flush()
         process_handler.stdin.close()
         process_handler.stdin = None
@@ -705,8 +524,6 @@ class Slapgrid(object):
     """
     Process a Computer Partition, depending on its state
     """
-    logger = logging.getLogger('ComputerPartitionProcessing')
-
     computer_partition_id = computer_partition.getId()
 
     # Sanity checks before processing
@@ -720,7 +537,7 @@ class Slapgrid(object):
          (computer_partition_id not in self.computer_partition_filter_list):
       return
 
-    logger.info('Processing Computer Partition %s...' % computer_partition_id)
+    self.logger.info('Processing Computer Partition %s.' % computer_partition_id)
 
     instance_path = os.path.join(self.instance_root, computer_partition_id)
 
@@ -742,13 +559,16 @@ class Slapgrid(object):
       # Try to process it anyway, it may need to be deleted.
       software_url = None
     try:
-      software_path = os.path.join(self.software_root,
-          getSoftwareUrlHash(software_url))
+      software_path = os.path.join(self.software_root, md5digest(software_url))
     except TypeError:
       # Problem with instance: SR URI not set.
       # Try to process it anyway, it may need to be deleted.
       software_path = None
 
+    self.logger.info('  Software URL: %s' % software_url)
+    self.logger.info('  Software path: %s' % software_path)
+    self.logger.info('  Instance path: %s' % instance_path)
+
     periodicity = self.maximum_periodicity
     if software_path:
       # Get periodicity from periodicity file if not forced
@@ -759,8 +579,7 @@ class Slapgrid(object):
             periodicity = int(open(periodicity_path).read())
           except ValueError:
             os.remove(periodicity_path)
-            exception = traceback.format_exc()
-            logger.error(exception)
+            self.logger.error(traceback.format_exc())
 
     # Check if timestamp from server is more recent than local one.
     # If not: it's not worth processing this partition (nothing has
@@ -786,8 +605,7 @@ class Slapgrid(object):
               os.remove(timestamp_path)
         except ValueError:
           os.remove(timestamp_path)
-          exception = traceback.format_exc()
-          logger.error(exception)
+          self.logger.error(traceback.format_exc())
 
     local_partition = Partition(
       software_path=software_path,
@@ -802,7 +620,8 @@ class Slapgrid(object):
       server_url=self.master_url,
       software_release_url=software_url,
       certificate_repository_path=self.certificate_repository_path,
-      buildout=self.buildout)
+      buildout=self.buildout,
+      logger=self.logger)
 
     computer_partition_state = computer_partition.getState()
     if computer_partition_state == COMPUTER_PARTITION_STARTED_STATE:
@@ -826,15 +645,14 @@ class Slapgrid(object):
       try:
         computer_partition.stopped()
       except (SystemExit, KeyboardInterrupt):
-        exception = traceback.format_exc()
-        computer_partition.error(exception)
+        computer_partition.error(traceback.format_exc(), logger=self.logger)
         raise
       except Exception:
         pass
     else:
       error_string = "Computer Partition %r has unsupported state: %s" % \
         (computer_partition_id, computer_partition_state)
-      computer_partition.error(error_string)
+      computer_partition.error(error_string, logger=self.logger)
       raise NotImplementedError(error_string)
 
     # If partition has been successfully processed, write timestamp
@@ -849,7 +667,6 @@ class Slapgrid(object):
     """
     Try to filter valid partitions to be processed from free partitions.
     """
-    logger = logging.getLogger('ComputerPartitionProcessing')
     filtered_computer_partition_list = []
     for computer_partition in computer_partition_list:
       try:
@@ -880,32 +697,29 @@ class Slapgrid(object):
 
       # Send log before exiting
       except (SystemExit, KeyboardInterrupt):
-        exception = traceback.format_exc()
-        computer_partition.error(exception)
+        computer_partition.error(traceback.format_exc(), logger=self.logger)
         raise
 
       # Buildout failed: send log but don't print it to output (already done)
-      except BuildoutFailedError, exception:
+      except BuildoutFailedError as exc:
         try:
-          computer_partition.error(exception)
+          computer_partition.error(exc, logger=self.logger)
         except (SystemExit, KeyboardInterrupt):
           raise
         except Exception:
-          exception = traceback.format_exc()
-          logger.error('Problem during reporting error, continuing:\n' +
-            exception)
+          self.logger.error('Problem during reporting error, continuing:\n%s' %
+                            traceback.format_exc())
 
       # For everything else: log it, send it, continue.
-      except Exception as exception:
-        logger.error(traceback.format_exc())
+      except Exception as exc:
+        self.logger.error(traceback.format_exc())
         try:
-          computer_partition.error(exception)
+          computer_partition.error(exc, logger=self.logger)
         except (SystemExit, KeyboardInterrupt):
           raise
         except Exception:
-          exception = traceback.format_exc()
-          logger.error('Problem during reporting error, continuing:\n' +
-            exception)
+          self.logger.error('Problem during reporting error, continuing:\n%s' %
+                            traceback.format_exc())
 
     return filtered_computer_partition_list
 
@@ -913,8 +727,7 @@ class Slapgrid(object):
     """
     Will start supervisord and process each Computer Partition.
     """
-    logger = logging.getLogger('ComputerPartitionProcessing')
-    logger.info('Processing computer partitions...')
+    self.logger.info('Processing computer partitions...')
     # Prepares environment
     self.checkEnvironmentAndCreateStructure()
     self._launchSupervisord()
@@ -938,48 +751,44 @@ class Slapgrid(object):
 
       # Send log before exiting
       except (SystemExit, KeyboardInterrupt):
-        exception = traceback.format_exc()
-        computer_partition.error(exception)
+        computer_partition.error(traceback.format_exc(), logger=self.logger)
         raise
 
-      except Slapgrid.PromiseError as exception:
+      except Slapgrid.PromiseError as exc:
         clean_run_promise = False
         try:
-          logger.error(exception)
-          computer_partition.error(exception)
+          self.logger.error(exc)
+          computer_partition.error(exc, logger=self.logger)
         except (SystemExit, KeyboardInterrupt):
           raise
         except Exception:
-          exception = traceback.format_exc()
-          logger.error('Problem during reporting error, continuing:\n' +
-            exception)
+          self.logger.error('Problem during reporting error, continuing:\n%s' %
+                            traceback.format_exc())
 
       # Buildout failed: send log but don't print it to output (already done)
-      except BuildoutFailedError, exception:
+      except BuildoutFailedError as exc:
         clean_run = False
         try:
-          computer_partition.error(exception)
+          computer_partition.error(exc, logger=self.logger)
         except (SystemExit, KeyboardInterrupt):
           raise
         except Exception:
-          exception = traceback.format_exc()
-          logger.error('Problem during reporting error, continuing:\n' +
-            exception)
+          self.logger.error('Problem during reporting error, continuing:\n%s' %
+                            traceback.format_exc())
 
       # For everything else: log it, send it, continue.
-      except Exception as exception:
+      except Exception as exc:
         clean_run = False
-        logger.error(traceback.format_exc())
+        self.logger.error(traceback.format_exc())
         try:
-          computer_partition.error(exception)
+          computer_partition.error(exc, logger=self.logger)
         except (SystemExit, KeyboardInterrupt):
           raise
         except Exception:
-          exception = traceback.format_exc()
-          logger.error('Problem during reporting error, continuing:\n' +
-            exception)
+          self.logger.error('Problem during reporting error, continuing:\n%s' %
+                            traceback.format_exc())
 
-    logger.info("Finished computer partitions.")
+    self.logger.info('Finished computer partitions.')
 
     # Return success value
     if not clean_run:
@@ -991,9 +800,6 @@ class Slapgrid(object):
 
   def validateXML(self, to_be_validated, xsd_model):
     """Validates a given xml file"""
-
-    logger = logging.getLogger('XMLValidating')
-
     #We retrieve the xsd model
     xsd_model = StringIO.StringIO(xsd_model)
     xmlschema_doc = etree.parse(xsd_model)
@@ -1001,10 +807,10 @@ class Slapgrid(object):
 
     try:
       document = etree.fromstring(to_be_validated)
-    except (etree.XMLSyntaxError, etree.DocumentInvalid) as e:
-      logger.info('Failed to parse this XML report :  %s\n%s' % \
-        (to_be_validated, _formatXMLError(e)))
-      logger.error(_formatXMLError(e))
+    except (etree.XMLSyntaxError, etree.DocumentInvalid) as exc:
+      self.logger.info('Failed to parse this XML report :  %s\n%s' % \
+        (to_be_validated, _formatXMLError(exc)))
+      self.logger.error(_formatXMLError(exc))
       return False
 
     if xmlschema.validate(document):
@@ -1033,17 +839,16 @@ class Slapgrid(object):
     for computer_partition_usage in computer_partition_usage_list:
       try:
         root = etree.fromstring(computer_partition_usage.usage)
-      except UnicodeError, e:
-        self.logger.info("Failed to read %s." % (
-            computer_partition_usage.usage))
+      except UnicodeError as exc:
+        self.logger.info("Failed to read %s." % computer_partition_usage.usage)
         self.logger.error(UnicodeError)
-        raise UnicodeError("Failed to read %s: %s" % (computer_partition_usage.usage, e))
-      except (etree.XMLSyntaxError, etree.DocumentInvalid) as e:
+        raise UnicodeError("Failed to read %s: %s" % (computer_partition_usage.usage, exc))
+      except (etree.XMLSyntaxError, etree.DocumentInvalid) as exc:
         self.logger.info("Failed to parse %s." % (computer_partition_usage.usage))
-        self.logger.error(e)
-        raise _formatXMLError(e)
-      except Exception, e:
-        raise Exception("Failed to generate XML report: %s" % e)
+        self.logger.error(exc)
+        raise _formatXMLError(exc)
+      except Exception as exc:
+        raise Exception("Failed to generate XML report: %s" % exc)
 
       for movement in root.findall('movement'):
         xml.append('<movement>')
@@ -1067,8 +872,7 @@ class Slapgrid(object):
 
     slap_computer_usage = self.slap.registerComputer(self.computer_id)
     computer_partition_usage_list = []
-    logger = logging.getLogger('UsageReporting')
-    logger.info("Aggregating and sending usage reports...")
+    self.logger.info('Aggregating and sending usage reports...')
 
     #We retrieve XSD models
     try:
@@ -1135,26 +939,26 @@ class Slapgrid(object):
           #stat sys call to get statistics informations
           uid = stat_info.st_uid
           gid = stat_info.st_gid
-          kw = dict(stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
           process_handler = SlapPopen(invocation_list,
-            preexec_fn=lambda: dropPrivileges(uid, gid),
+                                      preexec_fn=lambda: dropPrivileges(uid, gid, logger=self.logger),
                                       cwd=os.path.join(instance_path, 'etc', 'report'),
-            env=None, **kw)
+                                      env=None,
+                                      stdout=subprocess.PIPE,
+                                      stderr=subprocess.STDOUT,
+                                      logger=self.logger)
           if process_handler.returncode is None:
             process_handler.kill()
           if process_handler.returncode != 0:
             clean_run = False
             failed_script_list.append("Script %r failed." % script)
-            logger.warning("Failed to run %r" % invocation_list)
+            self.logger.warning('Failed to run %r' % invocation_list)
           if len(failed_script_list):
-            computer_partition.error('\n'.join(failed_script_list))
+            computer_partition.error('\n'.join(failed_script_list), logger=self.logger)
       # Whatever happens, don't stop processing other instances
       except Exception:
-        computer_partition_id = computer_partition.getId()
-        exception = traceback.format_exc()
-        issue = "Cannot run usage script(s) for %r: %s" % (
-            computer_partition_id, exception)
-        logger.info(issue)
+        self.logger.info('Cannot run usage script(s) for %r: %s' % (
+                            computer_partition.getId(),
+                            traceback.format_exc()))
 
     #Now we loop through the different computer partitions to report
     report_usage_issue_cp_list = []
@@ -1170,20 +974,17 @@ class Slapgrid(object):
           filename_list = os.listdir(dir_reports)
         else:
           filename_list = []
-        #logger.debug('name List %s' % filename_list)
-        usage = ''
+        #self.logger.debug('name List %s' % filename_list)
 
         for filename in filename_list:
 
           file_path = os.path.join(dir_reports, filename)
           if os.path.exists(file_path):
-            usage_file = open(file_path, 'r')
-            usage = usage_file.read()
-            usage_file.close()
+            usage = open(file_path, 'r').read()
 
             #We check the validity of xml content of each reports
             if not self.validateXML(usage, partition_consumption_model):
-              logger.info('WARNING: The XML file %s generated by slapreport is '
+              self.logger.info('WARNING: The XML file %s generated by slapreport is '
                                'not valid - This report is left as is at %s where you can '
                                'inspect what went wrong ' % (filename, dir_reports))
               # Warn the SlapOS Master that a partition generates corrupted xml
@@ -1195,7 +996,7 @@ class Slapgrid(object):
               computer_partition_usage_list.append(computer_partition_usage)
               filename_delete_list.append(filename)
           else:
-            logger.debug("Usage report %r not found, ignored" % file_path)
+            self.logger.debug('Usage report %r not found, ignored' % file_path)
 
         #After sending the aggregated file we remove all the valid xml reports
         for filename in filename_delete_list:
@@ -1203,15 +1004,14 @@ class Slapgrid(object):
 
       # Whatever happens, don't stop processing other instances
       except Exception:
-        computer_partition_id = computer_partition.getId()
-        exception = traceback.format_exc()
-        issue = "Cannot run usage script(s) for %r: %s" % (
-            computer_partition_id, exception)
-        logger.info(issue)
+        self.logger.info('Cannot run usage script(s) for %r: %s' % (
+                            computer_partition.getId(),
+                            traceback.format_exc()))
 
     for computer_partition_usage in computer_partition_usage_list:
-      logger.info('computer_partition_usage_list : %s - %s' % \
-        (computer_partition_usage.usage, computer_partition_usage.getId()))
+      self.logger.info('computer_partition_usage_list: %s - %s' % (
+                          computer_partition_usage.usage,
+                          computer_partition_usage.getId()))
 
     #If there is, at least, one report
     if computer_partition_usage_list != []:
@@ -1219,22 +1019,21 @@ class Slapgrid(object):
         #We generate the final XML report with asXML method
         computer_consumption = self.asXML(computer_partition_usage_list)
 
-        logger.info('Final xml report : %s' % computer_consumption)
+        self.logger.info('Final xml report: %s' % computer_consumption)
 
         #We test the XML report before sending it
         if self.validateXML(computer_consumption, computer_consumption_model):
-          logger.info('XML file generated by asXML is valid')
+          self.logger.info('XML file generated by asXML is valid')
           slap_computer_usage.reportUsage(computer_consumption)
         else:
-          logger.info('XML file generated by asXML is not valid !')
+          self.logger.info('XML file generated by asXML is not valid !')
           raise ValueError('XML file generated by asXML is not valid !')
       except Exception:
-        computer_partition_id = computer_partition.getId()
-        exception = traceback.format_exc()
-        issue = "Cannot report usage for %r: %s" % (computer_partition_id,
-          exception)
-        logger.info(issue)
-        computer_partition.error(issue)
+        issue = "Cannot report usage for %r: %s" % (
+                    computer_partition.getId(),
+                    traceback.format_exc())
+        self.logger.info(issue)
+        computer_partition.error(issue, logger=self.logger)
         report_usage_issue_cp_list.append(computer_partition_id)
 
     for computer_partition in computer_partition_list:
@@ -1243,11 +1042,11 @@ class Slapgrid(object):
           computer_partition_id = computer_partition.getId()
           try:
              software_url = computer_partition.getSoftwareRelease().getURI()
-             software_path = os.path.join(self.software_root,
-                 getSoftwareUrlHash(software_url))
+             software_path = os.path.join(self.software_root, md5digest(software_url))
           except (NotFoundError, TypeError):
             software_url = None
             software_path = None
+
           local_partition = Partition(
             software_path=software_path,
             instance_path=os.path.join(self.instance_root,
@@ -1263,42 +1062,40 @@ class Slapgrid(object):
             software_release_url=software_url,
             certificate_repository_path=self.certificate_repository_path,
             buildout=self.buildout,
-            )
+            logger=self.logger)
           local_partition.stop()
           try:
             computer_partition.stopped()
           except (SystemExit, KeyboardInterrupt):
-            exception = traceback.format_exc()
-            computer_partition.error(exception)
+            computer_partition.error(traceback.format_exc(), logger=self.logger)
             raise
           except Exception:
             pass
           if computer_partition.getId() in report_usage_issue_cp_list:
-            logger.info('Ignoring destruction of %r, as not report usage was '
-              'sent' % computer_partition.getId())
+            self.logger.info('Ignoring destruction of %r, as no report usage was sent' %
+                                computer_partition.getId())
             continue
           local_partition.destroy()
         except (SystemExit, KeyboardInterrupt):
-          exception = traceback.format_exc()
-          computer_partition.error(exception)
+          computer_partition.error(traceback.format_exc(), logger=self.logger)
           raise
         except Exception:
           clean_run = False
-          exception = traceback.format_exc()
-          computer_partition.error(exception)
-          logger.error(exception)
+          exc = traceback.format_exc()
+          computer_partition.error(exc, logger=self.logger)
+          self.logger.error(exc)
         try:
           computer_partition.destroyed()
-        except slap.NotFoundError:
-          logger.debug('Ignored slap error while trying to inform about '
+        except NotFoundError:
+          self.logger.debug('Ignored slap error while trying to inform about '
                             'destroying not fully configured Computer Partition %r' %
                                 computer_partition.getId())
         except ServerError as server_error:
-          logger.debug('Ignored server error while trying to inform about '
-              'destroying Computer Partition %r. Error is :\n%r' %
+          self.logger.debug('Ignored server error while trying to inform about '
+                            'destroying Computer Partition %r. Error is:\n%r' %
                                 (computer_partition.getId(), server_error.args[0]))
 
-    logger.info("Finished usage reports.")
+    self.logger.info('Finished usage reports.')
 
     # Return success value
     if not clean_run:

slapos/grid/svcbackend.py

@@ -28,18 +28,16 @@
 #
 ##############################################################################
 
-from supervisor import xmlrpc
 import time
-from utils import SlapPopen
-import logging
 import os
 import sys
 import xmlrpclib
-from optparse import OptionParser
-import ConfigParser
 import socket as socketlib
 import subprocess
 
+from supervisor import xmlrpc
+from slapos.grid.utils import SlapPopen
+
 
 def getSupervisorRPC(socket):
   supervisor_transport = xmlrpc.SupervisorTransport('', '',
@@ -49,20 +47,19 @@ def getSupervisorRPC(socket):
   return getattr(server_proxy, 'supervisor')
 
 
-def launchSupervisord(socket, configuration_file):
-  logger = logging.getLogger('SVCBackend')
-  supervisor = getSupervisorRPC(socket)
+def launchSupervisord(socket, configuration_file, logger):
   if os.path.exists(socket):
     trynum = 1
     while trynum < 6:
       try:
+        supervisor = getSupervisorRPC(socket)
         status = supervisor.getState()
       except xmlrpclib.Fault as e:
         if e.faultCode == 6 and e.faultString == 'SHUTDOWN_STATE':
           logger.info('Supervisor in shutdown procedure, will check again later.')
           trynum += 1
           time.sleep(2 * trynum)
-      except Exception:
+      except Exception as e:
         # In case if there is problem with connection, assume that supervisord
         # is not running and try to run it
         break
@@ -90,10 +87,14 @@ def launchSupervisord(socket, configuration_file):
                                 env={},
                                 executable=sys.executable,
                                 stdout=subprocess.PIPE,
-                                stderr=subprocess.STDOUT)
+                                stderr=subprocess.STDOUT,
+                                logger=logger)
+
   result = supervisord_popen.communicate()[0]
-  if supervisord_popen.returncode == 0:
-    logger.info('Supervisord command invoked with: %s' % result)
+  if supervisord_popen.returncode:
+    logger.warning('Supervisord unknown problem: %s' % result)
+    return
+
   try:
     default_timeout = socketlib.getdefaulttimeout()
     current_timeout = 1
@@ -101,13 +102,14 @@ def launchSupervisord(socket, configuration_file):
     while trynum < 6:
       try:
         socketlib.setdefaulttimeout(current_timeout)
+        supervisor = getSupervisorRPC(socket)
         status = supervisor.getState()
         if status['statename'] == 'RUNNING' and status['statecode'] == 1:
           return
         logger.warning('Wrong status name %(statename)r and code '
           '%(statecode)r, trying again' % status)
         trynum += 1
-        except Exception:
+      except Exception as e:
         current_timeout = 5 * trynum
         trynum += 1
       else:
@@ -116,59 +118,3 @@ def launchSupervisord(socket, configuration_file):
     logger.warning('Issue while checking supervisord.')
   finally:
     socketlib.setdefaulttimeout(default_timeout)
-
-  else:
-    log_message = 'Supervisord unknown problem: %s' % result
-    logger.warning(log_message)
-
-
-def getOptionDict(*argument_tuple):
-  usage = """
-Typical usage:
- * %prog CONFIGURATION_FILE [arguments passed to supervisor]
-
-""".strip()
-
-  parser = OptionParser(usage=usage)
-
-  # Parses arguments
-  if argument_tuple:
-    (argument_option_instance, argument_list) = parser.parse_args(list(argument_tuple))
-  else:
-    # No arguments given to entry point : we parse sys.argv.
-    (argument_option_instance, argument_list) = parser.parse_args()
-
-  if not argument_list:
-    parser.error("Configuration file is obligatory. Consult documentation by "
-        "calling with -h.")
-  configuration_file = argument_list[0]
-  if not os.path.exists(configuration_file):
-    parser.error("Could not read configuration file : %s" % configuration_file)
-
-  slapgrid_configuration = ConfigParser.SafeConfigParser()
-  slapgrid_configuration.read(configuration_file)
-
-  # Merges the two dictionnaries
-  option_dict = dict(slapgrid_configuration.items("slapos"))
-  # Supervisord configuration location
-  option_dict.setdefault('supervisord_configuration_path',
-                         os.path.join(option_dict['instance_root'], 'etc', 'supervisord.conf'))
-  # Supervisord socket
-  option_dict.setdefault('supervisord_socket',
-                         os.path.join(option_dict['instance_root'], 'supervisord.socket'))
-  return option_dict, argument_list[1:]
-
-
-def supervisorctl(*argument_tuple):
-  option_dict, args = getOptionDict(*argument_tuple)
-  import supervisor.supervisorctl
-  launchSupervisord(option_dict['supervisord_socket'],
-      option_dict['supervisord_configuration_path'])
-  supervisor.supervisorctl.main(args=['-c',
-    option_dict['supervisord_configuration_path']] + args)
-
-def supervisord(*argument_tuple):
-  option_dict, _ = getOptionDict(*argument_tuple)
-  launchSupervisord(option_dict['supervisord_socket'],
-      option_dict['supervisord_configuration_path'])
-

slapos/grid/utils.py

 # -*- coding: utf-8 -*-
+# vim: set et sts=2:
 ##############################################################################
 #
 # Copyright (c) 2010, 2011, 2012 Vifib SARL and Contributors.
@@ -27,21 +28,20 @@
 #
 ##############################################################################
 
-import logging
+import grp
 import hashlib
 import os
 import pkg_resources
+import pwd
 import stat
 import subprocess
 import sys
-import pwd
-import grp
-from exception import BuildoutFailedError, WrongPermissionError
-from hashlib import md5
+
+from slapos.grid.exception import BuildoutFailedError, WrongPermissionError
 
 # Such umask by default will create paths with full permission
 # for user, non writable by group and not accessible by others
-SAFE_UMASK = 027
+SAFE_UMASK = 0o27
 
 PYTHON_ENVIRONMENT_REMOVE_LIST = [
   'PYTHONHOME',
@@ -94,6 +94,7 @@ class SlapPopen(subprocess.Popen):
   log.
   """
   def __init__(self, *args, **kwargs):
+    logger = kwargs.pop('logger')
     kwargs.update(stdin=subprocess.PIPE)
     if sys.platform == 'cygwin' and kwargs.get('env') == {}:
       kwargs['env'] = None
@@ -102,25 +103,22 @@ class SlapPopen(subprocess.Popen):
     self.stdin.close()
     self.stdin = None
 
-    logger = logging.getLogger('SlapProcessManager')
     # XXX-Cedric: this algorithm looks overkill for simple logging.
     output_lines = []
     while True:
       line = self.stdout.readline()
-      if line == '' and self.poll() != None:
+      if line == '' and self.poll() is not None:
         break
       output_lines.append(line)
-      if line[-1:] == '\n':
-        line = line[:-1]
-      logger.info(line)
+      logger.info(line.rstrip('\n'))
     self.output = ''.join(output_lines)
 
-def getSoftwareUrlHash(url):
-  return md5(url).hexdigest()
+
+def md5digest(url):
+  return hashlib.md5(url).hexdigest()
 
 
-def getCleanEnvironment(home_path='/tmp'):
-  logger = logging.getLogger('CleanEnvironment')
+def getCleanEnvironment(logger, home_path='/tmp'):
   changed_env = {}
   removed_env = []
   env = os.environ.copy()
@@ -137,46 +135,41 @@ def getCleanEnvironment(home_path='/tmp'):
   return env
 
 
-def setRunning(pid_file):
+def setRunning(logger, pidfile):
   """Creates a pidfile. If a pidfile already exists, we exit"""
-  logger = logging.getLogger('Slapgrid')
-  if os.path.exists(pid_file):
-    # Pid file is present
+  # XXX might use http://code.activestate.com/recipes/577911-context-manager-for-a-daemon-pid-file/
+  if os.path.exists(pidfile):
     try:
-      pid = int(open(pid_file, 'r').readline())
+      pid = int(open(pidfile, 'r').readline())
     except ValueError:
       pid = None
     # XXX This could use psutil library.
-    if pid is not None and os.path.exists("/proc/%s" % pid):
-      # In case process is present, ignore.
+    if pid and os.path.exists("/proc/%s" % pid):
       logger.info('New slapos process started, but another slapos '
                   'process is aleady running with pid %s, exiting.' % pid)
       sys.exit(10)
-    logger.info('Existing pid file %r was stale one, overwritten' % pid_file)
+    logger.info('Existing pid file %r was stale, overwritten' % pidfile)
   # Start new process
-  write_pid(pid_file)
+  write_pid(logger, pidfile)
 
 
-def setFinished(pid_file):
+def setFinished(pidfile):
   try:
-    os.remove(pid_file)
+    os.remove(pidfile)
   except OSError:
     pass
 
 
-def write_pid(pid_file):
-  logger = logging.getLogger('Slapgrid')
-  pid = os.getpid()
+def write_pid(logger, pidfile):
   try:
-    f = open(pid_file, 'w')
-    f.write('%s' % pid)
-    f.close()
+    with open(pidfile, 'w') as fout:
+      fout.write('%s' % os.getpid())
   except (IOError, OSError):
-    logger.critical('slapgrid could not write pidfile %s' % pid_file)
+    logger.critical('slapgrid could not write pidfile %s' % pidfile)
     raise
 
 
-def dropPrivileges(uid, gid):
+def dropPrivileges(uid, gid, logger):
   """Drop privileges to uid, gid if current uid is 0
 
   Do tests to check if dropping was successful and that no system call is able
@@ -184,14 +177,13 @@ def dropPrivileges(uid, gid):
 
   Does nothing in case if uid and gid are not 0
   """
-  logger = logging.getLogger('dropPrivileges')
   # XXX-Cedric: remove format / just do a print, otherwise formatting is done
   # twice
   current_uid, current_gid = os.getuid(), os.getgid()
   if uid == 0 or gid == 0:
     raise OSError('Dropping privileges to uid = %r or ' \
                                       'gid = %r is too dangerous' % (uid, gid))
-  if not(current_uid == 0 and current_gid == 0):
+  if current_uid or current_gid:
     logger.debug('Running as uid = %r, gid = %r, dropping not needed and not '
         'possible' % (current_uid, current_gid))
     return
@@ -209,7 +201,7 @@ def dropPrivileges(uid, gid):
                             uid, gid, group_list)
   new_uid, new_gid, new_group_list = os.getuid(), os.getgid(), os.getgroups()
   if not (new_uid == uid and new_gid == gid and set(new_group_list) == group_list):
-    raise OSError('%s new_uid = %r and new_gid = %r and ' \
+    raise OSError('%s new_uid = %r and new_gid = %r and '
                   'new_group_list = %r which is fatal.'
                     % (message_pre,
                        new_uid,
@@ -236,18 +228,16 @@ def dropPrivileges(uid, gid):
   logger.debug('Succesfully dropped privileges to uid=%r gid=%r' % (uid, gid))
 
 
-def bootstrapBuildout(path, buildout=None,
+def bootstrapBuildout(path, logger, buildout=None,
                       additional_buildout_parametr_list=None):
   if additional_buildout_parametr_list is None:
     additional_buildout_parametr_list = []
-  logger = logging.getLogger('BuildoutManager')
   # Reads uid/gid of path, launches buildout with thoses privileges
   stat_info = os.stat(path)
   uid = stat_info.st_uid
   gid = stat_info.st_gid
 
   invocation_list = [sys.executable, '-S']
-  kw = dict()
   if buildout is not None:
     invocation_list.append(buildout)
     invocation_list.extend(additional_buildout_parametr_list)
@@ -274,25 +264,26 @@ def bootstrapBuildout(path, buildout=None,
     logger.debug('Set umask from %03o to %03o' % (umask, SAFE_UMASK))
     logger.debug('Invoking: %r in directory %r' % (' '.join(invocation_list),
       path))
-    kw.update(stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
     process_handler = SlapPopen(invocation_list,
-            preexec_fn=lambda: dropPrivileges(uid, gid),
-            cwd=path, **kw)
+                                preexec_fn=lambda: dropPrivileges(uid, gid, logger=logger),
+                                cwd=path,
+                                stdout=subprocess.PIPE,
+                                stderr=subprocess.STDOUT,
+                                logger=logger)
     if process_handler.returncode is None or process_handler.returncode != 0:
       message = 'Failed to run buildout profile in directory %r' % (path)
       logger.error(message)
       raise BuildoutFailedError('%s:\n%s\n' % (message, process_handler.output))
-  except OSError as error:
-    raise BuildoutFailedError(error)
+  except OSError as exc:
+    raise BuildoutFailedError(exc)
   finally:
     old_umask = os.umask(umask)
     logger.debug('Restore umask from %03o to %03o' % (old_umask, umask))
 
 
-def launchBuildout(path, buildout_binary,
+def launchBuildout(path, buildout_binary, logger,
                    additional_buildout_parametr_list=None):
   """ Launches buildout."""
-  logger = logging.getLogger('BuildoutManager')
   if additional_buildout_parametr_list is None:
     additional_buildout_parametr_list = []
   # Reads uid/gid of path, launches buildout with thoses privileges
@@ -300,9 +291,7 @@ def launchBuildout(path, buildout_binary,
   uid = stat_info.st_uid
   gid = stat_info.st_gid
   # Extract python binary to prevent shebang size limit
-  file = open(buildout_binary, 'r')
-  line = file.readline()
-  file.close()
+  line = open(buildout_binary, 'r').readline()
   invocation_list = []
   if line.startswith('#!'):
     line = line[2:]
@@ -316,42 +305,44 @@ def launchBuildout(path, buildout_binary,
     logger.debug('Set umask from %03o to %03o' % (umask, SAFE_UMASK))
     logger.debug('Invoking: %r in directory %r' % (' '.join(invocation_list),
       path))
-    kw = dict(stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
     process_handler = SlapPopen(invocation_list,
-            preexec_fn=lambda: dropPrivileges(uid, gid), cwd=path,
-            env=getCleanEnvironment(pwd.getpwuid(uid).pw_dir), **kw)
+                                preexec_fn=lambda: dropPrivileges(uid, gid, logger=logger),
+                                cwd=path,
+                                env=getCleanEnvironment(logger=logger,
+                                                        home_path=path),
+                                stdout=subprocess.PIPE,
+                                stderr=subprocess.STDOUT,
+                                logger=logger)
     if process_handler.returncode is None or process_handler.returncode != 0:
       message = 'Failed to run buildout profile in directory %r' % (path)
       logger.error(message)
       raise BuildoutFailedError('%s:\n%s\n' % (message, process_handler.output))
-  except OSError as error:
-    raise BuildoutFailedError(error)
+  except OSError as exc:
+    raise BuildoutFailedError(exc)
   finally:
     old_umask = os.umask(umask)
     logger.debug('Restore umask from %03o to %03o' % (old_umask, umask))
 
 
-def updateFile(file_path, content, mode='0600'):
+def updateFile(file_path, content, mode=0o600):
   """Creates an executable with "content" as content."""
   altered = False
   if not (os.path.isfile(file_path)) or \
      not(hashlib.md5(open(file_path).read()).digest() ==\
          hashlib.md5(content).digest()):
+    with open(file_path, 'w') as fout:
+      fout.write(content)
     altered = True
-    file_file = open(file_path, 'w')
-    file_file.write(content)
-    file_file.flush()
-    file_file.close()
   os.chmod(file_path, stat.S_IREAD | stat.S_IWRITE | stat.S_IEXEC)
-  if oct(stat.S_IMODE(os.stat(file_path).st_mode)) != mode:
-    os.chmod(file_path, int(mode, 8))
+  if stat.S_IMODE(os.stat(file_path).st_mode) != mode:
+    os.chmod(file_path, mode)
     altered = True
   return altered
 
 
 def updateExecutable(executable_path, content):
   """Creates an executable with "content" as content."""
-  return updateFile(executable_path, content, '0700')
+  return updateFile(executable_path, content, 0o700)
 
 
 def createPrivateDirectory(path):
@@ -359,8 +350,8 @@ def createPrivateDirectory(path):
   if not os.path.isdir(path):
     os.mkdir(path)
   os.chmod(path, stat.S_IREAD | stat.S_IWRITE | stat.S_IEXEC)
-  permission = oct(stat.S_IMODE(os.stat(path).st_mode))
-  if permission not in ('0700'):
-    raise WrongPermissionError('Wrong permissions in %s ' \
-                                        ': is %s, should be 0700'
+  permission = stat.S_IMODE(os.stat(path).st_mode)
+  if permission != 0o700:
+    raise WrongPermissionError('Wrong permissions in %s: ' \
+                               'is 0%o, should be 0700'
                                % (path, permission))

slapos/proxy/__init__.py

 # -*- coding: utf-8 -*-
+# vim: set et sts=2:
 ##############################################################################
 #
 # Copyright (c) 2010, 2011, 2012 Vifib SARL and Contributors.
@@ -27,113 +28,38 @@
 #
 ##############################################################################
 
-import os
-import sys
-from optparse import OptionParser, Option
 import logging
-import logging.handlers
-import ConfigParser
 
 
-class Parser(OptionParser):
-  """
-  Parse all arguments.
-  """
-  def __init__(self, usage=None, version=None):
-    """
-    Initialize all options possibles.
-    """
-    OptionParser.__init__(self, usage=usage, version=version,
-                          option_list=[
-      Option("-l", "--log_file",
-             help="The path to the log file used by the script.",
-             type=str),
-      Option("-v", "--verbose",
-             default=False,
-             action="store_true",
-             help="Verbose output."),
-      Option("-c", "--console",
-             default=False,
-             action="store_true",
-             help="Console output."),
-      Option("-u", "--database-uri",
-             type=str,
-             help="URI for sqlite database"),
-    ])
+class ProxyConfig(object):
+  def __init__(self, logger):
+    self.logger = logger
 
-  def check_args(self):
-    """
-    Check arguments
-    """
-    (options, args) = self.parse_args()
-    if len(args) != 1:
-      self.error("Incorrect number of arguments")
-
-    return options, args[0]
-
-class Config:
-  def setConfig(self, option_dict, configuration_file_path):
-    """
-    Set options given by parameters.
-    """
+  def mergeConfig(self, args, configp):
     # Set options parameters
-    for option, value in option_dict.__dict__.items():
+    for option, value in args.__dict__.items():
       setattr(self, option, value)
 
-    # Load configuration file
-    configuration_parser = ConfigParser.SafeConfigParser()
-    configuration_parser.read(configuration_file_path)
     # Merges the arguments and configuration
     for section in ("slapproxy", "slapos"):
-      configuration_dict = dict(configuration_parser.items(section))
+      configuration_dict = dict(configp.items(section))
       for key in configuration_dict:
         if not getattr(self, key, None):
           setattr(self, key, configuration_dict[key])
 
-    # set up logging
-    self.logger = logging.getLogger("slapproxy")
-    self.logger.setLevel(logging.INFO)
-    if self.console:
-      self.logger.addHandler(logging.StreamHandler())
 
+  def setConfig(self):
     if not self.database_uri:
       raise ValueError('database-uri is required.')
-    if self.log_file:
-      if not os.path.isdir(os.path.dirname(self.log_file)):
-        # fallback to console only if directory for logs does not exists and
-        # continue to run
-        raise ValueError('Please create directory %r to store %r log file' % (
-          os.path.dirname(self.log_file), self.log_file))
-      else:
-        file_handler = logging.FileHandler(self.log_file)
-        file_handler.setFormatter(logging.Formatter("%(asctime)s - %(name)s - %(levelname)s - %(message)s"))
-        self.logger.addHandler(file_handler)
-        self.logger.info('Configured logging to file %r' % self.log_file)
-
-    self.logger.info("Started.")
-    if self.verbose:
-      self.logger.setLevel(logging.DEBUG)
-      self.logger.debug("Verbose mode enabled.")
-
-def run(config):
-  from views import app
-  app.config['computer_id'] = config.computer_id
-  app.config['DATABASE_URI'] = config.database_uri
-  app.run(host=config.host, port=int(config.port))
 
-def main():
-  "Run default configuration."
-  usage = "usage: %s [options] CONFIGURATION_FILE" % sys.argv[0]
 
-  try:
-    # Parse arguments
-    config = Config()
-    config.setConfig(*Parser(usage=usage).check_args())
+def do_proxy(conf):
+  from slapos.proxy.views import app
+  for handler in conf.logger.handlers:
+    app.logger.addHandler(handler)
+  app.logger.setLevel(logging.INFO)
+  app.config['computer_id'] = conf.computer_id
+  app.config['DATABASE_URI'] = conf.database_uri
+  app.run(host=conf.host, port=int(conf.port))
 
-    run(config)
-    return_code = 0
-  except SystemExit, err:
-    # Catch exception raise by optparse
-    return_code = err
 
-  sys.exit(return_code)

slapos/register/register.py

 # -*- coding: utf-8 -*-
+# vim: set et sts=2:
 ##############################################################################
 #
 # Copyright (c) 2012 Vifib SARL and Contributors. All Rights Reserved.
@@ -26,12 +27,11 @@
 #
 ##############################################################################
 
+# XXX dry_run will happily register a new node on the slapos master. Isn't it supposed to be no-op?
+
 
-import base64
 import ConfigParser
-from getpass import getpass
-import logging
-from optparse import OptionParser, Option
+import getpass
 import os
 import shutil
 import stat
@@ -40,119 +40,40 @@ import tempfile
 import urllib2
 
 
-class SlapError(Exception):
-  """
-  Slap error
-  """
-  def __init__(self, message):
-    self.msg = message
-
-class UsageError(SlapError):
-  pass
-
-class ExecError(SlapError):
-  pass
+def authenticate(request, login, password):
+  auth = '%s:%s' % (login, password)
+  authheader = 'Basic %s' % auth.encode('base64').rstrip()
+  request.add_header('Authorization', authheader)
 
-class Parser(OptionParser):
-  """
-  Parse all arguments.
-  """
-  def __init__(self, usage=None, version=None):
-    """
-    Initialize all options possibles.
-    """
-    OptionParser.__init__(self, usage=usage, version=version,
-                          option_list=[
-      Option("--interface-name",
-             help="Interface name to access internet",
-             default='eth0',
-             type=str),
-      Option("--master-url",
-             help="URL of SlapOS master",
-             default='https://slap.vifib.com',
-             type=str),
-      Option("--master-url-web",
-             help="URL of SlapOS Master webservice to register certificates",
-             default='https://www.slapos.org',
-             type=str),
-      Option("--partition-number",
-             help="Number of partition on computer",
-             default='10',
-             type=int),
-      Option("--ipv4-local-network",
-             help="Base of ipv4 local network",
-             default='10.0.0.0/16',
-             type=str),
-      Option("--ipv6-interface",
-             help="Interface name to get ipv6",
-             default='',
-             type=str),
-      Option("--login",
-             help="User login on SlapOS Master webservice",
-             default=None,
-             type=str),
-      Option("--password",
-             help="User password on SlapOs Master webservice",
-             default=None,
-             type=str),
-      Option("-t", "--create-tap",
-             help="""Will trigger creation of one virtual "tap" interface per \
-Partition and attach it to primary interface. Requires primary interface to be \
- a bridge. defaults to false. Needed to host virtual machines.""",
-             default=False,
-             action="store_true"),
-      Option("-n", "--dry-run",
-             help="Simulate the execution steps",
-             default=False,
-             action="store_true"),
-   ])
-
-  def check_args(self):
-    """
-    Check arguments
-    """
-    (options, args) = self.parse_args()
-    if len(args) != 1:
-      self.error("Incorrect number of arguments")
-    node_name = args[0]
 
-    if options.password != None and options.login == None :
-      self.error("Please enter your login with your password")
-
-
-    return options, node_name
-
-
-def get_login():
-  """Get user id and encode it for basic identification"""
-  login = raw_input("SlapOS Master Login: ")
-  password = getpass()
-  identification = base64.encodestring('%s:%s' % (login, password))[:-1]
-  return identification
-
-
-def check_login(identification, master_url_web):
+def check_credentials(url, login, password):
   """Check if logged correctly on SlapOS Master"""
-  request = urllib2.Request(master_url_web)
-  # Prepare header for basic authentification
-  authheader =  "Basic %s" % identification
-  request.add_header("Authorization", authheader)
-  home_page_url = urllib2.urlopen(request).read()
-  if 'Logout' in home_page_url:
-    return 1
-  else : return 0
+  request = urllib2.Request(url)
+  authenticate(request, login, password)
+  return 'Logout' in urllib2.urlopen(request).read()
 
 
-def get_certificates(identification, node_name, master_url_web):
+def get_certificates(master_url_web, node_name, login, password, logger):
   """Download certificates from SlapOS Master"""
   register_server_url = '/'.join([master_url_web, ("add-a-server/WebSection_registerNewComputer?dialog_id=WebSection_viewServerInformationDialog&dialog_method=WebSection_registerNewComputer&title={}&object_path=/erp5/web_site_module/hosting/add-a-server&update_method=&cancel_url=https%3A//www.vifib.net/add-a-server/WebSection_viewServerInformationDialog&Base_callDialogMethod=&field_your_title=Essai1&dialog_category=None&form_id=view".format(node_name))])
   request = urllib2.Request(register_server_url)
-  # Prepare header for basic authentification
-  authheader =  "Basic %s" % identification
-  request.add_header("Authorization", authheader)  
-  url = urllib2.urlopen(request)  
-  page = url.read()
-  return page
+  authenticate(request, login, password)
+
+  try:
+    req = urllib2.urlopen(request)
+  except urllib2.HTTPError as exc:
+    # raise a readable exception if the computer name is already used,
+    # instead of an opaque 500 Internal Error.
+    # this will not work with the new API.
+    if exc.getcode() == 500:
+      error_body = exc.read()
+      if 'Certificate still active.' in error_body:
+        logger.error('The node name "%s" is already in use. Please change the name, or revoke the active certificate if you want to replace the node.' % node_name)
+        sys.exit(1)
+    raise
+
+  return req.read()
+
 
 
 def parse_certificates(source):
@@ -161,7 +82,7 @@ def parse_certificates(source):
   c_end = source.find("</textarea>", c_start)
   k_start = source.find("-----BEGIN PRIVATE KEY-----")
   k_end = source.find("</textarea>", k_start)
-  return [source[c_start:c_end], source[k_start:k_end]]
+  return source[c_start:c_end], source[k_start:k_end]
 
 
 def get_computer_name(certificate):
@@ -170,50 +91,50 @@ def get_computer_name(certificate):
   i = certificate.find("/email", k)
   return certificate[k:i]
 
-def save_former_config(config):
+
+def save_former_config(conf):
   """Save former configuration if found"""
   # Check for config file in /etc/opt/slapos/
   if os.path.exists('/etc/opt/slapos/slapos.cfg'):
-    former_slapos_configuration = '/etc/opt/slapos'
-  else : former_slapos_configuration = 0
-  if former_slapos_configuration:
-    saved_slapos_configuration = former_slapos_configuration + '.old'
+    former = '/etc/opt/slapos'
+  else:
+    return
+
+  saved = former + '.old'
   while True:
-      if os.path.exists(saved_slapos_configuration):
-        print "Slapos configuration detected in %s" % saved_slapos_configuration
-        if saved_slapos_configuration[len(saved_slapos_configuration) - 1] != 'd' :
-          saved_slapos_configuration = saved_slapos_configuration[:len(saved_slapos_configuration) - 1] \
-              + str(int(saved_slapos_configuration[len(saved_slapos_configuration) - 1]) + 1 )
-        else :
-          saved_slapos_configuration += ".1"
-      else: break
-    config.logger.info("Former slapos configuration detected in %s moving to %s" % (former_slapos_configuration, saved_slapos_configuration))
-    shutil.move(former_slapos_configuration, saved_slapos_configuration)
+    if os.path.exists(saved):
+      print "Slapos configuration detected in %s" % saved
+      if saved[-1] != 'd':
+        saved = saved[:-1] + str(int(saved[-1]) + 1)
+      else:
+        saved += '.1'
+    else:
+        break
+  conf.logger.info("Former slapos configuration detected in %s moving to %s" % (former, saved))
+  shutil.move(former, saved)
+
 
 def get_slapos_conf_example():
   """
   Get slapos.cfg.example and return its path
   """
-  register_server_url = "http://git.erp5.org/gitweb/slapos.core.git/blob_plain/HEAD:/slapos.cfg.example"
-  request = urllib2.Request(register_server_url)
-  url = urllib2.urlopen(request)  
-  page = url.read()
+  request = urllib2.Request('http://git.erp5.org/gitweb/slapos.core.git/blob_plain/HEAD:/slapos.cfg.example')
+  req = urllib2.urlopen(request)
   _, path = tempfile.mkstemp()
-  slapos_cfg_example = open(path,'w')
-  slapos_cfg_example.write(page)
-  slapos_cfg_example.close()
+  with open(path, 'w') as fout:
+    fout.write(req.read())
   return path
 
 
-def slapconfig(config):
+def slapconfig(conf):
   """Base Function to configure slapos in /etc/opt/slapos"""
-  dry_run = config.dry_run
+  dry_run = conf.dry_run
   # Create slapos configuration directory if needed
-  slap_configuration_directory = os.path.normpath(config.slapos_configuration)
+  slap_conf_dir = os.path.normpath(conf.slapos_configuration)
 
   # Make sure everybody can read slapos configuration directory:
   # Add +x to directories in path
-  directory = os.path.dirname(slap_configuration_directory)
+  directory = os.path.dirname(slap_conf_dir)
   while True:
     if os.path.dirname(directory) == directory:
       break
@@ -221,100 +142,84 @@ def slapconfig(config):
     os.chmod(directory, os.stat(directory).st_mode | stat.S_IXGRP | stat.S_IRGRP | stat.S_IXOTH | stat.S_IROTH)
     directory = os.path.dirname(directory)
 
-  if not os.path.exists(slap_configuration_directory):
-    config.logger.info ("Creating directory: %s" % slap_configuration_directory)
+  if not os.path.exists(slap_conf_dir):
+    conf.logger.info("Creating directory: %s" % slap_conf_dir)
     if not dry_run:
-      os.mkdir(slap_configuration_directory, 0711)
+      os.mkdir(slap_conf_dir, 0o711)
 
-  user_certificate_repository_path = os.path.join(slap_configuration_directory,'ssl')
+  user_certificate_repository_path = os.path.join(slap_conf_dir, 'ssl')
   if not os.path.exists(user_certificate_repository_path):
-    config.logger.info ("Creating directory: %s" % user_certificate_repository_path)
+    conf.logger.info("Creating directory: %s" % user_certificate_repository_path)
     if not dry_run:
-      os.mkdir(user_certificate_repository_path, 0711)
+      os.mkdir(user_certificate_repository_path, 0o711)
 
   key_file = os.path.join(user_certificate_repository_path, 'key')
   cert_file = os.path.join(user_certificate_repository_path, 'certificate')
-  for (src, dst) in [(config.key, key_file), (config.certificate,
-      cert_file)]:
-    config.logger.info ("Copying to %r, and setting minimum privileges" % dst)
+  for src, dst in [
+          (conf.key, key_file),
+          (conf.certificate, cert_file)
+          ]:
+    conf.logger.info("Copying to %r, and setting minimum privileges" % dst)
     if not dry_run:
-      destination = open(dst,'w')
+      with open(dst, 'w') as destination:
         destination.write(''.join(src))
-      destination.close()
-      os.chmod(dst, 0600)
+      os.chmod(dst, 0o600)
       os.chown(dst, 0, 0)
 
-  certificate_repository_path = os.path.join(slap_configuration_directory, 'ssl', 'partition_pki')
+  certificate_repository_path = os.path.join(slap_conf_dir, 'ssl', 'partition_pki')
   if not os.path.exists(certificate_repository_path):
-    config.logger.info ("Creating directory: %s" % certificate_repository_path)
+    conf.logger.info("Creating directory: %s" % certificate_repository_path)
     if not dry_run:
-      os.mkdir(certificate_repository_path, 0711)
+      os.mkdir(certificate_repository_path, 0o711)
 
   # Put slapos configuration file
-  slap_configuration_file_location = os.path.join(slap_configuration_directory,
-                                                  'slapos.cfg')
-  config.logger.info ("Creating slap configuration: %s"
-                      % slap_configuration_file_location)
+  slap_conf_file = os.path.join(slap_conf_dir, 'slapos.cfg')
+  conf.logger.info("Creating slap configuration: %s" % slap_conf_file)
 
   # Get example configuration file
   slapos_cfg_example = get_slapos_conf_example()
-  configuration_example_parser = ConfigParser.RawConfigParser()
-  configuration_example_parser.read(slapos_cfg_example)  
+  new_configp = ConfigParser.RawConfigParser()
+  new_configp.read(slapos_cfg_example)
   os.remove(slapos_cfg_example)
 
-  # prepare slapos section
-  slaposconfig = dict(
-    computer_id=config.computer_id, master_url=config.master_url,
-    key_file=key_file, cert_file=cert_file,
-    certificate_repository_path=certificate_repository_path)
-  for key in slaposconfig:
-    configuration_example_parser.set('slapos', key, slaposconfig[key])
-
-  # prepare slapformat
-  slapformatconfig = dict(
-    interface_name=config.interface_name,
-    ipv4_local_network=config.ipv4_local_network,
-    partition_amount=config.partition_number,
-    create_tap=config.create_tap
-    )
-  for key in slapformatconfig :
-    configuration_example_parser.set('slapformat', key, slapformatconfig[key])
-
-  if not config.ipv6_interface == '':
-    configuration_example_parser.set('slapformat',
-                                     'ipv6_interface',
-                                     config.ipv6_interface)
+  for section, key, value in [
+          ('slapos', 'computer_id', conf.computer_id),
+          ('slapos', 'master_url', conf.master_url),
+          ('slapos', 'key_file', key_file),
+          ('slapos', 'cert_file', cert_file),
+          ('slapos', 'certificate_repository_path', certificate_repository_path),
+          ('slapformat', 'interface_name', conf.interface_name),
+          ('slapformat', 'ipv4_local_network', conf.ipv4_local_network),
+          ('slapformat', 'partition_amount', conf.partition_number),
+          ('slapformat', 'create_tap', conf.create_tap)
+          ]:
+    new_configp.set(section, key, value)
+
+  if conf.ipv6_interface:
+    new_configp.set('slapformat', 'ipv6_interface', conf.ipv6_interface)
 
   if not dry_run:
-    slap_configuration_file = open(slap_configuration_file_location, "w")
-    configuration_example_parser.write(slap_configuration_file)
+    with open(slap_conf_file, 'w') as fout:
+      new_configp.write(fout)
+
+  conf.logger.info("SlapOS configuration: DONE")
 
-  config.logger.info ("SlapOS configuration: DONE")
 
+class RegisterConfig(object):
+  """
+  Class containing all parameters needed for configuration
+  """
+
+  def __init__(self, logger):
+    self.logger = logger
 
-# Class containing all parameters needed for configuration
-class Config:
-  def setConfig(self, option_dict, node_name):
+  def setConfig(self, options):
     """
     Set options given by parameters.
     """
     # Set options parameters
-    for option, value in option_dict.__dict__.items():
+    for option, value in options.__dict__.items():
       setattr(self, option, value)
-    self.node_name = node_name
-
-    # Define logger for register
-    self.logger = logging.getLogger('Register')
-    self.logger.setLevel(logging.DEBUG)
-    # create console handler and set level to debug
-    self.ch = logging.StreamHandler()
-    self.ch.setLevel(logging.INFO)
-    # create formatter
-    self.formatter = logging.Formatter('%(levelname)s - %(message)s')
-    # add formatter to ch
-    self.ch.setFormatter(self.formatter)
-    # add ch to logger
-    self.logger.addHandler(self.ch)
 
   def COMPConfig(self, slapos_configuration, computer_id, certificate, key):
     self.slapos_configuration = slapos_configuration
@@ -328,66 +233,46 @@ class Config:
     self.logger.debug("Number of partition: %s" % self.partition_number)
     self.logger.info("Using Interface %s" % self.interface_name)
     self.logger.debug("Ipv4 sub network: %s" % self.ipv4_local_network)
-    self.logger.debug("Ipv6 Interface: %s" %self.ipv6_interface)
+    self.logger.debug("Ipv6 Interface: %s" % self.ipv6_interface)
+
+
+def gen_auth(conf):
+  ask = True
+  if conf.login:
+    if conf.password:
+      yield conf.login, conf.password
+      ask = False
+    else:
+      yield conf.login, getpass.getpass()
+  while ask:
+    yield raw_input('SlapOS Master Login: '), getpass.getpass()
+
 
-def register(config):
+def do_register(conf):
   """Register new computer on SlapOS Master and generate slapos.cfg"""
-  # Get User identification and check them
-  if config.login == None :
-    while True :
-      print ("Please enter your SlapOS Master login")
-      user_id = get_login()
-      if check_login(user_id, config.master_url_web):
+
+  for login, password in gen_auth(conf):
+    if check_credentials(conf.master_url_web, login, password):
       break
-      config.logger.warning ("Wrong login/password")
+    conf.logger.warning('Wrong login/password')
   else:
-    if config.password == None :
-      config.password = getpass()
-    user_id = base64.encodestring('%s:%s' % (config.login, config.password))[:-1]
-    if not check_login(user_id, config.master_url_web):
-      config.logger.error ("Wrong login/password")
     return 1
 
   # Get source code of page having certificate and key
-  certificate_key = get_certificates(user_id, config.node_name, config.master_url_web)
+  certificate_key = get_certificates(conf.master_url_web, conf.node_name, login, password, conf.logger)
   # Parse certificate and key and get computer id
-  certificate_key = parse_certificates(certificate_key)
-  certificate = certificate_key[0]
-  key = certificate_key[1]
+  certificate, key = parse_certificates(certificate_key)
   COMP = get_computer_name(certificate)
   # Getting configuration parameters
-  slapos_configuration = '/etc/opt/slapos/'
-  config.COMPConfig(slapos_configuration=slapos_configuration,
+  conf.COMPConfig(slapos_configuration='/etc/opt/slapos/',
                   computer_id=COMP,
-                   certificate = certificate,
-                   key = key
-                   )
+                  certificate=certificate,
+                  key=key)
   # Save former configuration
-  if not config.dry_run:
-    save_former_config(config)
+  if not conf.dry_run:
+    save_former_config(conf)
   # Prepare Slapos Configuration
-  slapconfig(config)
+  slapconfig(conf)
 
   print "Node has successfully been configured as %s." % COMP
   return 0
-
-def main():
-  "Run default configuration."
-  usage = "usage: slapos node %s NODE_NAME [options] " % sys.argv[0]
-  try:
-    # Parse arguments
-    config = Config()
-    config.setConfig(*Parser(usage=usage).check_args())
-    return_code = register(config)
-  except UsageError, err:
-    print >> sys.stderr, err.msg
-    print >> sys.stderr, "For help use --help"
-    return_code = 16
-  except ExecError, err:
-    print >> sys.stderr, err.msg
-    return_code = 16
-  except SystemExit, err:
-    # Catch exception raise by optparse
-    return_code = err
-
-  sys.exit(return_code)

slapos/slap/slap.py

@@ -34,8 +34,6 @@ __all__ = ["slap", "ComputerPartition", "Computer", "SoftwareRelease",
            "Supply", "OpenOrder", "NotFoundError", "Unauthorized",
            "ResourceNotReady", "ServerError"]
 
-from interface import slap as interface
-from xml_marshaller import xml_marshaller
 import httplib
 import logging
 import socket
@@ -43,9 +41,16 @@ import ssl
 import traceback
 import urllib
 import urlparse
+
 import zope.interface
+from interface import slap as interface
+from xml_marshaller import xml_marshaller
+
+fallback_logger = logging.getLogger(__name__)
+fallback_handler = logging.StreamHandler()
+fallback_logger.setLevel(logging.INFO)
+fallback_logger.addHandler(fallback_handler)
 
-log = logging.getLogger(__name__)
 
 DEFAULT_SOFTWARE_TYPE = 'RootSoftwareInstance'
 
@@ -144,7 +149,7 @@ class SoftwareRelease(SlapDocument):
     else:
       return self._software_release
 
-  def error(self, error_log):
+  def error(self, error_log, logger=None):
     try:
       # Does not follow interface
       self._connection_helper.POST('/softwareReleaseError', {
@@ -152,8 +157,7 @@ class SoftwareRelease(SlapDocument):
         'computer_id' : self.getComputerId(),
         'error_log': error_log})
     except Exception:
-      exception = traceback.format_exc()
-      log.error(exception)
+      (logger or fallback_logger).error(traceback.format_exc())
 
   def available(self):
     self._connection_helper.POST('/availableSoftwareRelease', {
@@ -417,15 +421,14 @@ class ComputerPartition(SlapRequester):
       'computer_partition_id': self.getId(),
       })
 
-  def error(self, error_log):
+  def error(self, error_log, logger=None):
     try:
       self._connection_helper.POST('/softwareInstanceError', {
         'computer_id': self._computer_id,
         'computer_partition_id': self.getId(),
         'error_log': error_log})
     except Exception:
-      exception = traceback.format_exc()
-      log.error(exception)
+      (logger or fallback_logger).error(traceback.format_exc())
 
   def bang(self, message):
     self._connection_helper.POST('/softwareInstanceBang', {
@@ -660,7 +663,7 @@ class slap:
     returns SoftwareRelease class object
     """
     return SoftwareRelease(software_release=software_release,
-      connection_helper=self._connection_helper,
+      connection_helper=self._connection_helper
     )
 
   def registerComputer(self, computer_guid):

slapos/tests/entry.py

@@ -28,7 +28,7 @@
 
 import os
 import shutil
-import slapos.entry as entry
+from slapos.cli_legacy import entry
 import sys
 import tempfile
 import unittest
@@ -126,15 +126,15 @@ class TestCall (BasicMixin, unittest.TestCase):
 
     options = ["--logfile /opt/slapos/logfile",
                "--pidfile /opt/slapos/pidfile"]
-    config = '/etc/opt/slapos/slapos.cfg'
+    config_path = '/etc/opt/slapos/slapos.cfg'
     try:
-      entry.call(fun, config=config, option=options)
+      entry.call(fun, config_path=config_path, option=options)
     except SystemExit, e:
       self.assertEqual(e[0], 0)
     self.assertNotEqual(original_sysargv, sys.argv)
     for x in options:
       self.assertTrue(x in " ".join(sys.argv))
-    self.assertEqual(config, sys.argv[1])
+    self.assertEqual(config_path, sys.argv[1])
 
   def test_config_and_missing_option_are_added(self):
     """
@@ -151,16 +151,16 @@ class TestCall (BasicMixin, unittest.TestCase):
       return 0
 
     options = [default_present_option, missing_option]
-    config = '/etc/opt/slapos/slapos.cfg'
+    config_path = '/etc/opt/slapos/slapos.cfg'
     try:
-      entry.call(fun, config=config, option=options)
+      entry.call(fun, config_path=config_path, option=options)
     except SystemExit, e:
       self.assertEqual(e[0], 0)
     self.assertNotEqual(original_sysargv, sys.argv)
     for x in (missing_option, present_option):
       self.assertTrue(x in " ".join(sys.argv))
     self.assertFalse(default_present_option in " ".join(sys.argv))
-    self.assertEqual(config, sys.argv[1])
+    self.assertEqual(config_path, sys.argv[1])
 
   def test_present_config_and_option_are_not_added(self):
     """
@@ -178,9 +178,9 @@ class TestCall (BasicMixin, unittest.TestCase):
       return 0
 
     options = [default_present_option]
-    config = '/etc/opt/slapos/slapos.cfg'
+    config_path = '/etc/opt/slapos/slapos.cfg'
     try:
-      entry.call(fun, config=config, option=options)
+      entry.call(fun, config_path=config_path, option=options)
     except SystemExit, e:
       self.assertEqual(e[0], 0)
 

slapos/tests/slap.py

@@ -24,12 +24,15 @@
 # Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA  02111-1307, USA.
 #
 ##############################################################################
-import xml_marshaller
+
+import httplib
+import os
 import unittest
 import urlparse
-import httplib
+
 import slapos.slap
-import os
+import xml_marshaller
+
 
 class UndefinedYetException(Exception):
   """To catch exceptions which are not yet defined"""

slapos/tests/slapformat.py

@@ -244,7 +244,9 @@ class TestComputer(SlapformatMixin):
   @unittest.skip("Not implemented")
   def test_construct_empty_prepared(self):
     computer = slapos.format.Computer('computer',
-      interface=slapos.format.Interface('bridge', '127.0.0.1/16'))
+      interface=slapos.format.Interface(logger=self.test_result,
+                                        name='bridge',
+                                        ipv4_local_network='127.0.0.1/16'))
     computer.instance_root = '/instance_root'
     computer.software_root = '/software_root'
     computer.construct()
@@ -263,7 +265,9 @@ class TestComputer(SlapformatMixin):
 
   def test_construct_empty_prepared_no_alter_user(self):
     computer = slapos.format.Computer('computer',
-      interface=slapos.format.Interface('bridge', '127.0.0.1/16'))
+      interface=slapos.format.Interface(logger=self.test_result,
+                                        name='bridge',
+                                        ipv4_local_network='127.0.0.1/16'))
     computer.instance_root = '/instance_root'
     computer.software_root = '/software_root'
     computer.construct(alter_user=False)
@@ -279,7 +283,9 @@ class TestComputer(SlapformatMixin):
   @unittest.skip("Not implemented")
   def test_construct_empty_prepared_no_alter_network(self):
     computer = slapos.format.Computer('computer',
-      interface=slapos.format.Interface('bridge', '127.0.0.1/16'))
+      interface=slapos.format.Interface(logger=self.test_result,
+                                        name='bridge',
+                                        ipv4_local_network='127.0.0.1/16'))
     computer.instance_root = '/instance_root'
     computer.software_root = '/software_root'
     computer.construct(alter_network=False)
@@ -298,7 +304,9 @@ class TestComputer(SlapformatMixin):
 
   def test_construct_empty_prepared_no_alter_network_user(self):
     computer = slapos.format.Computer('computer',
-      interface=slapos.format.Interface('bridge', '127.0.0.1/16'))
+      interface=slapos.format.Interface(logger=self.test_result,
+                                        name='bridge',
+                                        ipv4_local_network='127.0.0.1/16'))
     computer.instance_root = '/instance_root'
     computer.software_root = '/software_root'
     computer.construct(alter_network=False, alter_user=False)
@@ -315,7 +323,9 @@ class TestComputer(SlapformatMixin):
   @unittest.skip("Not implemented")
   def test_construct_prepared(self):
     computer = slapos.format.Computer('computer',
-      interface=slapos.format.Interface('bridge', '127.0.0.1/16'))
+      interface=slapos.format.Interface(logger=self.test_result,
+                                        name='bridge',
+                                        ipv4_local_network='127.0.0.1/16'))
     computer.instance_root = '/instance_root'
     computer.software_root = '/software_root'
     partition = slapos.format.Partition('partition', '/part_path',
@@ -359,7 +369,9 @@ class TestComputer(SlapformatMixin):
 
   def test_construct_prepared_no_alter_user(self):
     computer = slapos.format.Computer('computer',
-      interface=slapos.format.Interface('bridge', '127.0.0.1/16'))
+      interface=slapos.format.Interface(logger=self.test_result,
+                                        name='bridge',
+                                        ipv4_local_network='127.0.0.1/16'))
     computer.instance_root = '/instance_root'
     computer.software_root = '/software_root'
     partition = slapos.format.Partition('partition', '/part_path',
@@ -400,7 +412,9 @@ class TestComputer(SlapformatMixin):
   @unittest.skip("Not implemented")
   def test_construct_prepared_no_alter_network(self):
     computer = slapos.format.Computer('computer',
-      interface=slapos.format.Interface('bridge', '127.0.0.1/16'))
+      interface=slapos.format.Interface(logger=self.test_result,
+                                        name='bridge',
+                                        ipv4_local_network='127.0.0.1/16'))
     computer.instance_root = '/instance_root'
     computer.software_root = '/software_root'
     partition = slapos.format.Partition('partition', '/part_path',
@@ -440,7 +454,9 @@ class TestComputer(SlapformatMixin):
 
   def test_construct_prepared_no_alter_network_user(self):
     computer = slapos.format.Computer('computer',
-      interface=slapos.format.Interface('bridge', '127.0.0.1/16'))
+      interface=slapos.format.Interface(logger=self.test_result,
+                                        name='bridge',
+                                        ipv4_local_network='127.0.0.1/16'))
     computer.instance_root = '/instance_root'
     computer.software_root = '/software_root'
     partition = slapos.format.Partition('partition', '/part_path',

slapos/tests/slapgrid.py

@@ -25,25 +25,31 @@
 #
 ##############################################################################
 
-from slapos.grid import slapgrid
 import httplib
 import logging
 import os
-from random import random
+import random
 import shutil
 import signal
-import slapos.slap.slap
-import slapos.grid.utils
-from slapos.grid.watchdog import Watchdog, getWatchdogID
 import socket
 import sys
 import tempfile
 import textwrap
 import time
-import unittest
+import unittest2
 import urlparse
+
 import xml_marshaller
 
+import slapos.slap.slap
+import slapos.grid.utils
+from slapos.grid import slapgrid
+from slapos.cli_legacy.slapgrid import parseArgumentTupleAndReturnSlapgridObject
+from slapos.grid.utils import md5digest
+from slapos.grid.watchdog import Watchdog, getWatchdogID
+
+dummylogger = logging.getLogger()
+
 
 WATCHDOG_TEMPLATE = """#!%(python_path)s -S
 
@@ -112,10 +118,15 @@ class BasicMixin:
       'supervisord')
     self.usage_report_periodicity = 1
     self.buildout = None
-    self.grid = slapgrid.Slapgrid(self.software_root, self.instance_root,
-      self.master_url, self.computer_id, self.supervisord_socket,
+    self.grid = slapgrid.Slapgrid(self.software_root,
+                                  self.instance_root,
+                                  self.master_url,
+                                  self.computer_id,
+                                  self.supervisord_socket,
                                   self.supervisord_configuration_path,
-      self.buildout, develop=develop)
+                                  self.buildout,
+                                  develop=develop,
+                                  logger=logging.getLogger())
     # monkey patch buildout bootstrap
     def dummy(*args, **kw):
       pass
@@ -142,7 +153,7 @@ class BasicMixin:
     shutil.rmtree(self._tempdir, True)
 
 
-class TestBasicSlapgridCP(BasicMixin, unittest.TestCase):
+class TestBasicSlapgridCP(BasicMixin, unittest2.TestCase):
   def test_no_software_root(self):
     self.assertRaises(OSError, self.grid.processComputerPartitionList)
 
@@ -308,7 +319,7 @@ class ComputerForTest:
           return (200, {}, '')
 
       elif method == 'POST' and 'url' in parsed_qs:
-        # XXX hardcoded to first sofwtare release!
+        # XXX hardcoded to first software release!
         software = self.software_list[0]
         software.sequence.append(parsed_url.path)
         if parsed_url.path == 'buildingSoftwareRelease':
@@ -381,11 +392,11 @@ class InstanceForTest:
       os.mkdir(certificate_repository_path)
     self.cert_file = os.path.join(certificate_repository_path,
                                   "%s.crt" % self.name)
-    self.certificate = str(random())
+    self.certificate = str(random.random())
     open(self.cert_file, 'w').write(self.certificate)
     self.key_file = os.path.join(certificate_repository_path,
                                   "%s.key" % self.name)
-    self.key = str(random())
+    self.key = str(random.random())
     open(self.key_file, 'w').write(self.key)
 
 class SoftwareForTest:
@@ -400,8 +411,7 @@ class SoftwareForTest:
     self.software_root = software_root
     self.name = 'http://sr%s/' % name
     self.sequence = []
-    self.software_hash = \
-        slapos.grid.utils.getSoftwareUrlHash(self.name)
+    self.software_hash = md5digest(self.name)
     self.srdir = os.path.join(self.software_root, self.software_hash)
     self.requested_state = 'available'
     os.mkdir(self.srdir)
@@ -442,7 +452,7 @@ touch worked"""):
 
 
 
-class TestSlapgridCPWithMaster(MasterMixin, unittest.TestCase):
+class TestSlapgridCPWithMaster(MasterMixin, unittest2.TestCase):
 
   def test_nothing_to_do(self):
 
@@ -736,7 +746,7 @@ exit 1
     self.assertEqual('stopped', instance.state)
 
 
-class TestSlapgridCPWithMasterWatchdog(MasterMixin, unittest.TestCase):
+class TestSlapgridCPWithMasterWatchdog(MasterMixin, unittest2.TestCase):
 
   def setUp(self):
     MasterMixin.setUp(self)
@@ -911,7 +921,7 @@ class TestSlapgridCPWithMasterWatchdog(MasterMixin, unittest.TestCase):
       self.assertEqual(computer.sequence,[])
 
 
-class TestSlapgridCPPartitionProcessing (MasterMixin, unittest.TestCase):
+class TestSlapgridCPPartitionProcessing (MasterMixin, unittest2.TestCase):
 
   def test_partition_timestamp(self):
     computer = ComputerForTest(self.software_root,self.instance_root)
@@ -1298,7 +1308,7 @@ echo %s; echo %s; exit 42""" % (line1, line2))
     self.assertTrue("Failed to run buildout" in instance.error_log)
 
 
-class TestSlapgridUsageReport(MasterMixin, unittest.TestCase):
+class TestSlapgridUsageReport(MasterMixin, unittest2.TestCase):
   """
   Test suite about slapgrid-ur
   """
@@ -1500,7 +1510,7 @@ class TestSlapgridUsageReport(MasterMixin, unittest.TestCase):
 
 
 
-class TestSlapgridSoftwareRelease(MasterMixin, unittest.TestCase):
+class TestSlapgridSoftwareRelease(MasterMixin, unittest2.TestCase):
   def test_one_software_buildout_fail_is_correctly_logged(self):
     """
     1. We set up a software using a corrupted buildout
@@ -1521,7 +1531,7 @@ echo %s; echo %s; exit 42""" % (line1, line2))
     self.assertTrue(line2 in software.error_log)
     self.assertTrue("Failed to run buildout" in software.error_log)
 
-class SlapgridInitialization(unittest.TestCase):
+class SlapgridInitialization(unittest2.TestCase):
   """
   "Abstract" class setting setup and teardown for TestSlapgridArgumentTuple
   and TestSlapgridConfigurationFile.
@@ -1571,7 +1581,7 @@ class TestSlapgridArgumentTuple(SlapgridInitialization):
     """
       Raises if the argument list if empty and without configuration file.
     """
-    parser = slapgrid.parseArgumentTupleAndReturnSlapgridObject
+    parser = parseArgumentTupleAndReturnSlapgridObject
     # XXX: SystemExit is too generic exception, it is only known that
     #      something is wrong
     self.assertRaises(SystemExit, parser, *())
@@ -1581,7 +1591,7 @@ class TestSlapgridArgumentTuple(SlapgridInitialization):
       Check if we can have the slapgrid object returned with the minimum
       arguments.
     """
-    parser = slapgrid.parseArgumentTupleAndReturnSlapgridObject
+    parser = parseArgumentTupleAndReturnSlapgridObject
     return_list = parser(*self.default_arg_tuple)
     self.assertEquals(2, len(return_list))
 
@@ -1589,19 +1599,19 @@ class TestSlapgridArgumentTuple(SlapgridInitialization):
     """
       Raises if the  signature_private_key_file does not exists.
     """
-    parser = slapgrid.parseArgumentTupleAndReturnSlapgridObject
+    parser = parseArgumentTupleAndReturnSlapgridObject
     argument_tuple = ("--signature_private_key_file", "/non/exists/path") + \
                       self.default_arg_tuple
-    # XXX: SystemExit is too generic exception, it is only known that
-    #      something is wrong
-    self.assertRaises(SystemExit, parser, *argument_tuple)
+    self.assertRaisesRegexp(RuntimeError,
+                            "File '/non/exists/path' does not exist.",
+                            parser, *argument_tuple)
 
   def test_signature_private_key_file(self):
     """
       Check if the signature private key argument value is available on
       slapgrid object.
     """
-    parser = slapgrid.parseArgumentTupleAndReturnSlapgridObject
+    parser = parseArgumentTupleAndReturnSlapgridObject
     argument_tuple = ("--signature_private_key_file",
                       self.signature_key_file_descriptor.name) + \
                       self.default_arg_tuple
@@ -1613,7 +1623,7 @@ class TestSlapgridArgumentTuple(SlapgridInitialization):
     """
       Check if giving --all triggers "develop" option.
     """
-    parser = slapgrid.parseArgumentTupleAndReturnSlapgridObject
+    parser = parseArgumentTupleAndReturnSlapgridObject
     argument_tuple = ("--all",) + self.default_arg_tuple
     slapgrid_object = parser(*argument_tuple)[0]
     self.assertTrue(slapgrid_object.develop)
@@ -1623,7 +1633,7 @@ class TestSlapgridArgumentTuple(SlapgridInitialization):
       Check if not giving --all neither --develop triggers "develop"
       option to be False.
     """
-    parser = slapgrid.parseArgumentTupleAndReturnSlapgridObject
+    parser = parseArgumentTupleAndReturnSlapgridObject
     argument_tuple = self.default_arg_tuple
     slapgrid_object = parser(*argument_tuple)[0]
     self.assertFalse(slapgrid_object.develop)
@@ -1633,7 +1643,7 @@ class TestSlapgridArgumentTuple(SlapgridInitialization):
       Check if not giving --maximum-periodicity triggers "force_periodicity"
       option to be false.
     """
-    parser = slapgrid.parseArgumentTupleAndReturnSlapgridObject
+    parser = parseArgumentTupleAndReturnSlapgridObject
     argument_tuple = self.default_arg_tuple
     slapgrid_object = parser(*argument_tuple)[0]
     self.assertFalse(slapgrid_object.force_periodicity)
@@ -1642,7 +1652,7 @@ class TestSlapgridArgumentTuple(SlapgridInitialization):
     """
       Check if giving --maximum-periodicity triggers "force_periodicity" option.
     """
-    parser = slapgrid.parseArgumentTupleAndReturnSlapgridObject
+    parser = parseArgumentTupleAndReturnSlapgridObject
     argument_tuple = ("--maximum-periodicity","40") + self.default_arg_tuple
     slapgrid_object = parser(*argument_tuple)[0]
     self.assertTrue(slapgrid_object.force_periodicity)
@@ -1666,7 +1676,7 @@ upload-to-binary-cache-url-blacklist =
   http://2/bla
 """ % dict(fake_file=self.fake_file_descriptor.name))
     self.slapos_config_descriptor.seek(0)
-    slapgrid_object = slapgrid.parseArgumentTupleAndReturnSlapgridObject(
+    slapgrid_object = parseArgumentTupleAndReturnSlapgridObject(
         *self.default_arg_tuple)[0]
     self.assertEqual(
         slapgrid_object.upload_to_binary_cache_url_blacklist,
@@ -1694,7 +1704,7 @@ download-from-binary-cache-url-blacklist =
   http://2/bla
 """ % dict(fake_file=self.fake_file_descriptor.name))
     self.slapos_config_descriptor.seek(0)
-    slapgrid_object = slapgrid.parseArgumentTupleAndReturnSlapgridObject(
+    slapgrid_object = parseArgumentTupleAndReturnSlapgridObject(
         *self.default_arg_tuple)[0]
     self.assertEqual(
         slapgrid_object.upload_to_binary_cache_url_blacklist,
@@ -1726,7 +1736,7 @@ download-from-binary-cache-url-blacklist =
   http://4/bla
 """ % dict(fake_file=self.fake_file_descriptor.name))
     self.slapos_config_descriptor.seek(0)
-    slapgrid_object = slapgrid.parseArgumentTupleAndReturnSlapgridObject(
+    slapgrid_object = parseArgumentTupleAndReturnSlapgridObject(
         *self.default_arg_tuple)[0]
     self.assertEqual(
         slapgrid_object.upload_to_binary_cache_url_blacklist,
@@ -1755,7 +1765,7 @@ binary-cache-url-blacklist =
   http://2/bla
 """ % dict(fake_file=self.fake_file_descriptor.name))
     self.slapos_config_descriptor.seek(0)
-    slapgrid_object = slapgrid.parseArgumentTupleAndReturnSlapgridObject(
+    slapgrid_object = parseArgumentTupleAndReturnSlapgridObject(
         *self.default_arg_tuple)[0]
     self.assertEqual(
         slapgrid_object.upload_to_binary_cache_url_blacklist,
@@ -1767,7 +1777,7 @@ binary-cache-url-blacklist =
     )
 
 
-class TestSlapgridCPWithMasterPromise(MasterMixin, unittest.TestCase):
+class TestSlapgridCPWithMasterPromise(MasterMixin, unittest2.TestCase):
   def test_one_failing_promise(self):
     computer = ComputerForTest(self.software_root,self.instance_root)
     instance = computer.instance_list[0]
@@ -1822,7 +1832,7 @@ exit 127""" % {'worked_file': worked_file})
                      slapos.grid.slapgrid.SLAPGRID_PROMISE_FAIL)
     self.assertTrue(os.path.isfile(worked_file))
 
-    self.assertEqual(instance.error_log, 'Error')
+    self.assertEqual(instance.error_log[-5:], 'Error')
     self.assertTrue(instance.error)
     self.assertIsNone(instance.state)
 

slapos/tests/slapobject.py

@@ -25,12 +25,14 @@
 #
 ##############################################################################
 
+import logging
+import os
+import unittest
+
 from slapos.grid import SlapObject
 from slapos.grid import utils
 from slapos.grid import networkcache
 from slapos.tests.slapgrid import BasicMixin
-import os
-import unittest
 
 
 class FakeCallAndRead:
@@ -93,6 +95,7 @@ class TestSoftwareSlapObject(BasicMixin, unittest.TestCase):
             url='http://example.com/software.cfg',
             software_root=self.software_root,
             buildout=self.buildout,
+            logger=logging.getLogger(),
             signature_private_key_file='/signature/private/key_file',
             upload_cache_url='http://example.com/uploadcache',
             upload_dir_url='http://example.com/uploaddir',
@@ -129,7 +132,8 @@ class TestSoftwareSlapObject(BasicMixin, unittest.TestCase):
     software = SlapObject.Software(
             url='http://example.com/software.cfg',
             software_root=self.software_root,
-            buildout=self.buildout)
+            buildout=self.buildout,
+            logger=logging.getLogger())
 
     software.install()
 
@@ -161,6 +165,7 @@ class TestSoftwareSlapObject(BasicMixin, unittest.TestCase):
             url='http://example.com/software.cfg',
             software_root=self.software_root,
             buildout=self.buildout,
+            logger=logging.getLogger(),
             signature_private_key_file='/signature/private/key_file',
             upload_cache_url='http://example.com/uploadcache',
             upload_dir_url='http://example.com/uploaddir',
@@ -192,6 +197,7 @@ class TestSoftwareSlapObject(BasicMixin, unittest.TestCase):
             url='http://example.com/software.cfg',
             software_root=self.software_root,
             buildout=self.buildout,
+            logger=logging.getLogger(),
             signature_private_key_file='/signature/private/key_file',
             upload_cache_url='http://example.com/uploadcache',
             upload_dir_url='http://example.com/uploaddir',

slapos/tests/slapproxy.py

@@ -27,6 +27,7 @@
 #
 ##############################################################################
 
+import ConfigParser
 import os
 import logging
 import shutil
@@ -86,14 +87,16 @@ database_uri = %(tempdir)s/lib/proxy.db
     """
     Set config for slapproxy and start it
     """
-    config = slapos.proxy.Config()
-    config.setConfig(*(ProxyOption(self.proxy_db),
-                       self.slapos_cfg))
+    conf = slapos.proxy.ProxyConfig(logger=logging.getLogger())
+    configp = ConfigParser.SafeConfigParser()
+    configp.read(self.slapos_cfg)
+    conf.mergeConfig(ProxyOption(self.proxy_db), configp)
+    conf.setConfig()
     views.app.config['TESTING'] = True
     views.app.config['computer_id'] = self.computer_id
     views.app.config['DATABASE_URI'] = self.proxy_db
-    views.app.config['HOST'] = config.host
-    views.app.config['port'] = config.port
+    views.app.config['HOST'] = conf.host
+    views.app.config['port'] = conf.port
     self.app = views.app.test_client()
 
   def add_free_partition (self, partition_amount):