Skip to content

S
slapos.core
Commit 5f0924ff authored by Cédric de Saint Martin's avatar Cédric de Saint Martin
Browse files
Options

Merge branch 'cliff'

parents 92980bc2 f469ae7d
Show whitespace changes
Inline Side-by-side
Showing with 3344 additions and 2054 deletions
documentation/slapos.usage.rst deleted 100644 → 0
View file @ 92980bc2
=========================
SlapOS command line usage
=========================
Note:
-----
* Default configuration file of "Node" commands (slapos node, slapos supervisor) if not explicitly defined as first argument is:
/etc/opt/slapos/slapos.cfg or value of SLAPOS_CONFIGURATION environment variable if exists.
* Default configuration file of "Client" commands (slapos request, slapos supply, ...) if not explicitly defined as first argument is:
~/.slapos/slapos.cfg or value of SLAPOS_CLIENT_CONFIGURATION environment variable if exists.
* Default log file for Node commands is /var/log/[slapos-node-software.log | slapos-node-instance.log | slapos-node-report.log]. This one requires working log in slapgrid, currently log/console is a total mess.
* Default pid file for Node commands is: /var/run/[slapos-node-software.pid | slapos-node-instance.pid | slapos-node-report.pid].
* Default SlapOS Master is http://www.vifib.net. It can be changed by altering configuration files.
General commands
----------------
slapos
~~~~~~
Display help/usage.
SlapOS Client commands
----------------------
Those commands are used by clients (as human beings or programs) to manage their own instances.
slapos request
~~~~~~~~~~~~~~
Usage:
slapos request [slapos_configuration] <reference> <software_alias | software-url> [--node id=<computer guid> region=<region> network-type=<network> | location/to/node.json] [--configuration foo=value1 bar=value2 | location/to/configuration.json ] [--type type] [--slave]
Request an instance and get status and parameters of instance.
Examples:
* Request a wordpress instance named "mybeautifulinstance" on Node named "COMP-12345":
slapos request mybeautifulinstance wordpress --node id=COMP-12345
* Request a kvm instance named "mykvm" on Node named "COMP-12345", specifying nbd-host and nbd-ip parameters:
slapos request mykvm kvm --node id=COMP-12345 --configuration nbd-host=debian.nbd.vifib.org nbd-port=1024
XXX Change in slaplib: allow to fetch instance params without changing anything. i.e we should do "slapos request myalreadyrequestedinstance" to fetch connection parameters without erasing previously defined instance parameters.
slapos search
~~~~~~~~~~~~~
Note: Not yet implemented.
Usage:
slapos search <search parameters ex. computer region, instance reference, source_section, etc.>
Returns visible instances matching search parameters.
slapos supply
~~~~~~~~~~~~~
Usage:
slapos supply <software | software_group> <computer_guid | commputer_group>
Ask installation of a software on a specific node or group of nodes. Nodes will then be ready to accept instances of specified software.
Examples:
* Ask installation of wordpress Software Release on COMP-12345:
slapos supply wordpress COMP-12345
slapos remove
~~~~~~~~~~~~~
Usage:
slapos remove <software | software_group> <computer_guid | commputer_group>
Ask Removal of a software on a specific node or group of nodes. Existing instances won't work anymore.
XXX "slapos autounsupply a.k.a slapos cleanup"
Examples:
* Ask installation of wordpress Software Release on COMP-12345:
slapos supply wordpress COMP-12345
slapos autosupply
~~~~~~~~~~~~~~~~~
Note: Not yet implemented.
Usage:
slapos autosupply <software | software_group> <computer_guid | computer_group>
Like "slapos suppply", but on-demand. Software will be (re)installed only when at least one instance of this software is requested. When no instance of this software is deployed on the node, it will be uninstalled.
slapos console
~~~~~~~~~~~~~~
Enter in a python console with slap library imported. See "Slapconsole" section to have detailed documentation.
slapos <stop|start|destroy>
~~~~~~~~~~~~~~~~~~~~~~~~~~~
Note: Not yet implemented.
Usage:
slapos <stop|start|destroy> <instance reference>
Ask start/stop/destruction of selected instance.
Example:
* Ask to stop "mywordpressinstance":
slapos stop mywordpressinstance
SlapOS Node commands
--------------------
This kind of commands are used to control the current SlapOS Node. Those commands are only useful for administrators of Nodes.
# XXX: add an environment variable for configuration file.
slapos node
~~~~~~~~~~~
Display status of Node and if not started, launch supervisor daemon.
Temporary note: equivalent of old slapgrid-supervisord + slapgrid-supervisorctl.
slapos node register
~~~~~~~~~~~~~~~~~~~~
Usage:
******
::
slapos node register <DESIRED NODE NAME> [--login LOGIN [--password PASSWORD]] [--interface-name INTERFACE] [--master-url URL <--master-url-web URL>] [--partition-number NUMBER] [--ipv4-local-network NETWORK] [--ipv6-interface INTERFACE] [--create-tap] [--dry-run]
If login is not provided, asks for user's vifib account then password.
Node will register itself, if not already done, to the SlapOS Master defined in configuration file, and will generate SlapOS configuration file.
XXX-Cedric should be like this: If desired node name is already taken, will raise an error.
XXX-Cedric: --master-url-web url will disappear in REST API. Currently, "register" uses SlapOS master web URL to register computer, so it needs the web URL (like http://www.vifib.net)
If Node is already registered (slapos.cfg and certificate already present), issues a warning, backups original configuration and creates new one.
XXX-Cedric should check for IPv6 in selected interface
Parameters:
***********
--login LOGIN Your SlapOS Master login. If not provided, asks it interactively.
--password PASSWORD Your SlapOS Master password. If not provided, asks it interactively. NOTE: giving password as parameter should be avoided for security reasons.
--interface-name INTERFACE Use interface as primary interface. IP of Partitions will be added to it. Defaults to "eth0".
--master-url URL URL of SlapOS Master REST API. defaults to "https://slap.vifib.com".
--master-url-web URL URL of SlapOS Master web access. defaults to "https://www.vifib.com".
--partition-number NUMBER Number of partitions that will have your SlapOS Node. defaults to "10".
--ipv4-local-network NETWORK Subnetwork used to assign local IPv4 addresses. It should be a not used network in order to avoid conflicts. defaults to 10.0.0.0/16.
-t, --create-tap Will trigger creation of one virtual "tap" interface per Partition and attach it to primary interface. Requires primary interface to be a bridge. defaults to false. Needed to host virtual machines.
-n, --dry-run Don't touch to anything in the filesystem. Used to debug.
Notes:
******
* "IPv6 interface" and "create tap" won't be put at all in the SlapOS Node configuration file if not explicitly written.
Examples:
*********
* Register computer named "mycomputer" to vifib::
slapos node register mycomputer
* Register computer named "mycomputer" to vifib using br0 as primary interface, tap0 as IPv6 interface and different local ipv4 subnet::
slapos node register mycomputer --interface-name br0 --ipv6-interface tap0 --ipv4-local-network 11.0.0.0/16
* Register computer named "mycomputer" to another SlapOS master accessible via https://www.myownslaposmaster.com, and SLAP webservice accessible via https://slap.myownslaposmaster.com (Note that this address should be the "slap" webservice URL, not web URL)::
slapos node register mycomputer --master-url https://slap.myownslaposmaster.com --master-url-web https://www.myownslaposmaster.com
XXX-Cedric : To be implemented
* Register computer named "mycomputer" to vifib, and ask to create tap interface to be able to host KVMs::
slapos node register mycomputer --create-tap
slapos node software
~~~~~~~~~~~~~~~~~~~~
Usage:
******
::
slapos node software [--logfile LOGFILE] [--verbose | -v] [--only_sr URL] [--all] [CONFIGURATION_FILE]
Run software installation/deletion.
Temporary note: equivalent of old slapgrid-sr.
# XXX: only_sr should be named ??? (process-only ?)
# XXX: add a "-vv", very verbose, option.
Parameters:
***********
--logfile LOGFILE If specified, will log as well output in the file located at FILE.
--only_sr URL Only process one specific Software Release that has been supplied on this Computer. If not supplied: do nothing.
--all Process all Software Releases, even already installed.
--verbose, -v Be more verbose.
Return values:
**************
(Among other standard Python return values)
0 Everything went fine
1 At least one software hasn't correctly been installed.
slapos node instance
~~~~~~~~~~~~~~~~~~~~
Usage:
******
::
slapos node instance [--logfile LOGFILE] [--verbose | -v] [--only_cp PARTITION] [--all] [CONFIGURATION_FILE]
Temporary note: equivalent of old slapgrid-cp.
Run instances deployment.
Parameters:
***********
--logfile LOGFILE If specified, will log as well output in the file located at FILE.
--only_cp PARTITION Only process one specific Computer Partition, if possible.
--all Force processing all Computer Partitions.
--verbose, -v Be more verbose.
Return values:
**************
(Among other standard Python return values)
0 Everything went fine
1 At least one instance hasn't correctly been processed.
2 At least one promise has failed.
slapos node report
~~~~~~~~~~~~~~~~~~
Usage:
******
::
slapos node report [--logfile LOGFILE] [--verbose | -v] [CONFIGURATION_FILE]
Run instance reports and garbage collection.
Temporary note: equivalent of old slapgrid-ur.
Parameters:
***********
--logfile LOGFILE If specified, will log as well output in the file located at FILE.
--verbose, -v Be more verbose.
Return values:
**************
(Among other standard Python return values)
0 Everything went fine
1 At least one instance hasn't correctly been processed.
slapos node <start|stop|restart|tail|status>
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Usage:
slapos node <start|stop|restart|tail|status> <instance>:[process]
Start/Stop/Restart/Show stdout/stderr of instance and/or process.
Examples:
* Start all processes of slappart3:
slapos node start slappart3:
* Stop only apache in slappart1:
slapos node stop slappart1:apache
* Show stdout/stderr of mysqld in slappart2:
slapos node tail slappart2:mysqld
slapos node supervisorctl
~~~~~~~~~~~~~~~~~~~~~~~~~
Usage:
slapos node supervisorctl
Enter into supervisor console.
slapos node supervisord
~~~~~~~~~~~~~~~~~~~~~~~
Usage:
slapos node supervisord
Launch, if not already launched, supervisor daemon.
slapos node log
~~~~~~~~~~~~~~~
Note: Not yet implemented.
Usage:
slapos node log <software|instance|report>
Display log.
documentation/source/conf.py
View file @ 5f0924ff
......@@ -22,7 +22,13 @@ sys.path.append(os.path.abspath('../../'))
# Add any Sphinx extension module names here, as strings. They can be extensions
# coming with Sphinx (named 'sphinx.ext.*') or your custom ones.
extensions = ['sphinx.ext.autodoc', 'sphinx.ext.intersphinx', 'sphinx.ext.ifconfig', 'repoze.sphinx.autointerface']
extensions = [
'sphinx.ext.autodoc',
'sphinx.ext.intersphinx',
'sphinx.ext.ifconfig',
'repoze.sphinx.autointerface',
'sphinxcontrib.programoutput'
]
# Add any paths that contain templates here, relative to this directory.
# templates_path = ['_templates']
......@@ -45,9 +51,9 @@ copyright = u'2011, Vifib'
# built documents.
#
# The short X.Y version.
version = '0.21-dev'
from slapos.version import version
# The full version, including alpha/beta/rc tags.
release = '0.21-dev'
release = version
# The language for content autogenerated by Sphinx. Refer to documentation
# for a list of supported languages.
......
documentation/source/slapos 0 → 120000
View file @ 5f0924ff
../../slapos/cli/entry.py
\ No newline at end of file
documentation/source/slapos.usage.rst 0 → 100644
View file @ 5f0924ff
=========================
SlapOS command line usage
=========================
Notes:
------
* Default SlapOS Master is http://www.slapos.org. It can be changed by altering configuration files or with the ``--master-url``
argument for commands that support it.
* Most commands take a configuration file parameter, provided as ``--cfg /path/to/file.cfg``.
If no such argument is provided:
* "node" commands read configuration from :file:`/etc/opt/slapos/slapos.cfg`, or the file referenced by the
``SLAPOS_CONFIGURATION`` environment variable.
* likewise, "client" commands (request, supply...) use :file:`~/.slapos/slapos.cfg`, or the ``SLAPOS_CLIENT_CONFIGURATION`` variable.
XXX TODO document 'alias' for software_url, software_group?, computer_group?
Common options
--------------
Without arguments, the ``slapos`` program lists all the available commands and common options.
.. program-output:: python slapos
The ``-q`` and ``-v`` options control the verbosity of console output (``-v``: DEBUG, default: INFO, ``-q``: WARNING).
Output to a logfile is not affected, and is the same as ``-v``.
SlapOS Client commands
----------------------
These commands are used by clients (as human beings or programs) to manage their own instances.
request
~~~~~~~
.. program-output:: python slapos help request
Examples
* Request a wordpress instance named "mybeautifulinstance" on Node named "COMP-12345"::
$ slapos request mybeautifulinstance wordpress --node id=COMP-12345
* Request a kvm instance named "mykvm" on Node named "COMP-12345", specifying nbd-host and nbd-ip parameters::
$ slapos request mykvm kvm --node id=COMP-12345 --configuration \
nbd-host=debian.nbd.vifib.net nbd-port=1024
XXX Change in slaplib: allow to fetch instance params without changing anything. i.e we should do "slapos request myalreadyrequestedinstance" to fetch connection parameters without erasing previously defined instance parameters.
..
search
~~~~~~
Note: Not yet implemented.
Usage:
slapos search <search parameters ex. computer region, instance reference, source_section, etc.>
Returns visible instances matching search parameters.
supply
~~~~~~
.. program-output:: python slapos help supply
Ask installation of a software on a specific node or group of nodes. Nodes will then be ready to accept instances of specified software.
Examples
* Ask installation of wordpress Software Release on COMP-12345::
$ slapos supply wordpress COMP-12345
remove
~~~~~~
.. program-output:: python slapos help remove
Ask Removal of a software from a specific node or group of nodes. Existing instances won't work anymore.
XXX "slapos autounsupply a.k.a slapos cleanup"
Examples
* Ask installation of wordpress Software Release on COMP-12345::
$ slapos supply wordpress COMP-12345
..
autosupply
~~~~~~~~~~
Note: Not yet implemented.
Usage:
slapos autosupply <software | software_group> <computer_guid | computer_group>
Like "slapos suppply", but on-demand. Software will be (re)installed only when at least one instance of this software is requested. When no instance of this software is deployed on the node, it will be uninstalled.
console
~~~~~~~
.. program-output:: python slapos help console
..
<stop|start|destroy>
~~~~~~~~~~~~~~~~~~~~
Note: Not yet implemented.
Usage:
slapos <stop|start|destroy> <instance reference>
Ask start/stop/destruction of selected instance.
Example:
* Ask to stop "mywordpressinstance"::
$ slapos stop mywordpressinstance
SlapOS Node commands
--------------------
This group of commands is used to control the current SlapOS Node. They are only useful to Node administrators.
node, node status
~~~~~~~~~~~~~~~~~
These are aliases for ``node supervisorctl status``.
It displays the status of the node, also running the supervisor daemon if needed.
.. program-output:: python slapos help node supervisorctl status
node register
~~~~~~~~~~~~~
.. program-output:: python slapos help node register
If login is not provided, asks for user's SlapOS Master account then password.
Node will register itself, if not already done, to the SlapOS Master defined in configuration file, and will generate SlapOS configuration file.
XXX-Cedric should be like this: If desired node name is already taken, will raise an error.
XXX-Cedric: --master-url-web url will disappear in REST API. Currently, "register" uses SlapOS master web URL to register computer, so it needs the web URL (like http://www.slapos.org)
If Node is already registered (slapos.cfg and certificate already present), issues a warning, backups original configuration and creates new one.
XXX-Cedric should check for IPv6 in selected interface
Parameters:
***********
--login LOGIN Your SlapOS Master login. If not provided, asks it interactively.
--password PASSWORD Your SlapOS Master password. If not provided, asks it interactively. NOTE: giving password as parameter should be avoided for security reasons.
--interface-name INTERFACE Use interface as primary interface. IP of Partitions will be added to it. Defaults to "eth0".
--master-url URL URL of SlapOS Master REST API. defaults to "https://slap.vifib.com".
--master-url-web URL URL of SlapOS Master web access. defaults to "https://www.vifib.com".
--partition-number NUMBER Number of partitions that will have your SlapOS Node. defaults to "10".
--ipv4-local-network NETWORK Subnetwork used to assign local IPv4 addresses. It should be a not used network in order to avoid conflicts. defaults to 10.0.0.0/16.
-t, --create-tap Will trigger creation of one virtual "tap" interface per Partition and attach it to primary interface. Requires primary interface to be a bridge. defaults to false. Needed to host virtual machines.
Notes:
******
* "IPv6 interface" and "create tap" won't be put at all in the SlapOS Node configuration file if not explicitly written.
Examples
* Register computer named "mycomputer" to SlapOS Master::
$ slapos node register mycomputer
* Register computer named "mycomputer" to SlapOS Master using br0 as primary interface, tap0 as IPv6 interface and different local ipv4 subnet::
$ slapos node register mycomputer --interface-name br0 --ipv6-interface tap0 \
--ipv4-local-network 11.0.0.0/16
* Register computer named "mycomputer" to another SlapOS master accessible via https://www.myownslaposmaster.com, and SLAP webservice accessible via https://slap.myownslaposmaster.com (Note that this address should be the "slap" webservice URL, not web URL)::
$ slapos node register mycomputer --master-url https://slap.myownslaposmaster.com \
--master-url-web https://www.myownslaposmaster.com
XXX-Cedric : To be implemented
* Register computer named "mycomputer" to SlapOS Master, and ask to create tap interface to be able to host KVMs::
$ slapos node register mycomputer --create-tap
node software
~~~~~~~~~~~~~
.. program-output:: python slapos help node software
Return values:
**************
(Among other standard Python return values)
* 0 Everything went fine
* 1 At least one software was not correctly installed.
node instance
~~~~~~~~~~~~~
.. program-output:: python slapos help node instance
Return values:
**************
(Among other standard Python return values)
* 0 Everything went fine
* 1 At least one instance was not correctly processed.
* 2 At least one promise has failed.
node report
~~~~~~~~~~~
.. program-output:: python slapos help node report
Run instance reports and garbage collection.
Return values:
**************
(Among other standard Python return values)
* 0 Everything went fine
* 1 At least one instance hasn't correctly been processed.
node start|stop|restart|tail|status
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
::
usage: slapos node <start|stop|restart|tail|status> [-h] [--cfg CFG] <instance>:[process]
Start/Stop/Restart/Show stdout/stderr of instance and/or process.
optional arguments:
-h, --help show this help message and exit
--cfg CFG SlapOS configuration file - defaults to
$SLAPOS_CONFIGURATION or /etc/opt/slapos/slapos.cfg
Examples
* Start all processes of slappart3::
$ slapos node start slappart3:
* Stop only apache in slappart1::
$ slapos node stop slappart1:apache
* Show stdout/stderr of mysqld in slappart2::
$ slapos node tail slappart2:mysqld
node supervisorctl
~~~~~~~~~~~~~~~~~~
.. program-output:: python slapos help node supervisorctl
node supervisord
~~~~~~~~~~~~~~~~
.. program-output:: python slapos help node supervisord
..
node log
~~~~~~~~
Note: Not yet implemented.
Usage:
slapos node log <software|instance|report>
Display log.
SlapOS Miscellaneous commands
-----------------------------
cache lookup
~~~~~~~~~~~~
.. program-output:: python slapos help cache lookup
Examples
* See if the wordpress Software Release is available in precompiled format for our distribution::
$ slapos cache lookup http://git.erp5.org/gitweb/slapos.git/blob_plain/refs/tags/slapos-0.156:/software/kvm/software.cfg
Software URL: http://git.erp5.org/gitweb/slapos.git/blob_plain/refs/tags/slapos-0.156:/software/kvm/software.cfg
MD5: 4410088e11f370503e9d78db4cfa4ec4
-------------
Available for:
distribution | version | id | compatible?
-----------------+--------------+----------------+-------------
CentOS | 6.3 | Final | no
Fedora | 17 | Beefy Miracle | no
Ubuntu | 12.04 | precise | yes
debian | 6.0.6 | | no
debian | 7.0 | | no
setup.py
View file @ 5f0924ff
......@@ -44,6 +44,7 @@ setup(name=name,
# XML
'zope.interface', # slap library implementes interfaces
'zc.buildout',
'cliff',
] + additional_install_requires,
extra_requires={'docs': ('Sphinx', 'repoze.sphinx.autointerface'),},
tests_require=[
......@@ -54,21 +55,43 @@ setup(name=name,
# accessing templates
entry_points={
'console_scripts': [
# One entry point to control them all
'slapos = slapos.entry:main',
'slapos-watchdog = slapos.grid.watchdog:main',
'slapproxy = slapos.proxy:main',
'slapproxy-query = slapos.proxy.query:main',
'slapproxy = slapos.cli_legacy.proxy_start:main',
'slapos = slapos.cli.entry:main',
# Deprecated entry points
'slapconsole = slapos.client:slapconsole',
'slapformat = slapos.format:main',
'slapgrid = slapos.grid.slapgrid:run',
'slapgrid-sr = slapos.grid.slapgrid:runSoftwareRelease',
'slapgrid-cp = slapos.grid.slapgrid:runComputerPartition',
'slapgrid-ur = slapos.grid.slapgrid:runUsageReport',
'slapgrid-supervisorctl = slapos.grid.svcbackend:supervisorctl',
'slapgrid-supervisord = slapos.grid.svcbackend:supervisord',
'bang = slapos.bang:main',
'slapconsole = slapos.cli_legacy.console:console',
'slapformat = slapos.cli_legacy.format:main',
'slapgrid-sr = slapos.cli_legacy.slapgrid:runSoftwareRelease',
'slapgrid-cp = slapos.cli_legacy.slapgrid:runComputerPartition',
'slapgrid-ur = slapos.cli_legacy.slapgrid:runUsageReport',
'slapgrid-supervisorctl = slapos.cli_legacy.svcbackend:supervisorctl',
'slapgrid-supervisord = slapos.cli_legacy.svcbackend:supervisord',
'bang = slapos.cli_legacy.bang:main',
],
'slapos.cli': [
# Utilities
'cache lookup = slapos.cli.cache:CacheLookupCommand',
# SlapOS Node commands
'node bang = slapos.cli.bang:BangCommand',
'node format = slapos.cli.format:FormatCommand',
'node register = slapos.cli.register:RegisterCommand',
'node supervisord = slapos.cli.supervisord:SupervisordCommand',
'node supervisorctl = slapos.cli.supervisorctl:SupervisorctlCommand',
'node status = slapos.cli.supervisorctl:SupervisorctlStatusCommand',
'node start = slapos.cli.supervisorctl:SupervisorctlStartCommand',
'node stop = slapos.cli.supervisorctl:SupervisorctlStopCommand',
'node restart = slapos.cli.supervisorctl:SupervisorctlRestartCommand',
'node tail = slapos.cli.supervisorctl:SupervisorctlTailCommand',
'node report = slapos.cli.slapgrid:ReportCommand',
'node software = slapos.cli.slapgrid:SoftwareCommand',
'node instance = slapos.cli.slapgrid:InstanceCommand',
# SlapOS client commands
'console = slapos.cli.console:ConsoleCommand',
'proxy start = slapos.cli.proxy_start:ProxyStartCommand',
'proxy show = slapos.cli.proxy_show:ProxyShowCommand',
'supply = slapos.cli.supply:SupplyCommand',
'remove = slapos.cli.remove:RemoveCommand',
'request = slapos.cli.request:RequestCommand',
]
},
test_suite="slapos.tests",
......
slapos/bang.py
View file @ 5f0924ff
# -*- coding: utf-8 -*-
# vim: set et sts=2:
##############################################################################
#
# Copyright (c) 2011, 2012 Vifib SARL and Contributors.
......@@ -28,31 +29,17 @@
##############################################################################
import slapos.slap.slap
import argparse
import ConfigParser
def main(*args):
parser = argparse.ArgumentParser()
parser.add_argument("-m", "--message", default='', help="Message for bang.")
parser.add_argument("configuration_file", nargs=1, type=argparse.FileType(),
help="SlapOS configuration file.")
if len(args) == 0:
argument = parser.parse_args()
else:
argument = parser.parse_args(list(args))
configuration_file = argument.configuration_file[0]
message = argument.message
# Loads config (if config specified)
configuration = ConfigParser.SafeConfigParser()
configuration.readfp(configuration_file)
computer_id = configuration.get('slapos', 'computer_id')
master_url = configuration.get('slapos', 'master_url')
if configuration.has_option('slapos', 'key_file'):
key_file = configuration.get('slapos', 'key_file')
def do_bang(configp, message):
computer_id = configp.get('slapos', 'computer_id')
master_url = configp.get('slapos', 'master_url')
if configp.has_option('slapos', 'key_file'):
key_file = configp.get('slapos', 'key_file')
else:
key_file = None
if configuration.has_option('slapos', 'cert_file'):
cert_file = configuration.get('slapos', 'cert_file')
if configp.has_option('slapos', 'cert_file'):
cert_file = configp.get('slapos', 'cert_file')
else:
cert_file = None
slap = slapos.slap.slap()
......
slapos/cache.py
View file @ 5f0924ff
# -*- coding: utf-8 -*-
import ast
import argparse
import ConfigParser
import hashlib
import json
import re
......@@ -16,29 +14,20 @@ from slapos.grid.distribution import patched_linux_distribution
def maybe_md5(s):
return re.match('[0-9a-f]{32}', s)
def cache_lookup():
parser = argparse.ArgumentParser()
parser.add_argument("configuration_file", help="SlapOS configuration file")
parser.add_argument("software_url", help="Your software url or MD5 hash")
args = parser.parse_args()
configuration_parser = ConfigParser.SafeConfigParser()
configuration_parser.read(args.configuration_file)
def do_lookup(configp, software_url):
cache_dir = configp.get('networkcache', 'download-binary-dir-url')
configuration_parser.items('networkcache')
cache_dir = configuration_parser.get('networkcache', 'download-binary-dir-url')
if maybe_md5(args.software_url):
md5 = args.software_url
if maybe_md5(software_url):
md5 = software_url
else:
md5 = hashlib.md5(args.software_url).hexdigest()
md5 = hashlib.md5(software_url).hexdigest()
try:
response = urllib2.urlopen('%s/%s' % (cache_dir, md5))
except urllib2.HTTPError as e:
if e.code == 404:
print 'Object not in cache: %s' % args.software_url
print 'Object not in cache: %s' % software_url
else:
print 'Error during cache lookup: %s (%s)' % (e.code, e.reason)
sys.exit(10)
......@@ -67,4 +56,3 @@ def cache_lookup():
for os in ostable:
compatible = 'yes' if networkcache.os_matches(os, linux_distribution) else 'no'
print '%-16s | %12s | %s | %s' % (os[0], os[1], os[2].center(14), compatible)
slapos/cli/bang.py 0 → 100644
View file @ 5f0924ff
# -*- coding: utf-8 -*-
import logging
from slapos.cli.config import ConfigCommand
from slapos.bang import do_bang
class BangCommand(ConfigCommand):
"""
request update on all partitions
"""
log = logging.getLogger('bang')
def get_parser(self, prog_name):
ap = super(BangCommand, self).get_parser(prog_name)
ap.add_argument('-m', '--message',
help='Message for bang')
return ap
def take_action(self, args):
configp = self.fetch_config(args)
do_bang(configp, args.message)
slapos/cli/cache.py 0 → 100644
View file @ 5f0924ff
# -*- coding: utf-8 -*-
import logging
from slapos.cli.config import ConfigCommand
from slapos.cache import do_lookup
class CacheLookupCommand(ConfigCommand):
"""
perform a query to the networkcache
You can provide either a complete URL to the software release,
or a corresponding MD5 hash value.
The command will report which OS distribution/version have a binary
cache of the software release, and which ones are compatible
with the OS you are currently running.
"""
log = logging.getLogger('cache-lookup')
def get_parser(self, prog_name):
ap = super(CacheLookupCommand, self).get_parser(prog_name)
ap.add_argument('software_url',
help='Your software url or MD5 hash')
return ap
def take_action(self, args):
configp = self.fetch_config(args)
do_lookup(configp, args.software_url)
slapos/cli/command.py 0 → 100644
View file @ 5f0924ff
# -*- coding: utf-8 -*-
import argparse
import cliff
class Command(cliff.command.Command):
def get_parser(self, prog_name):
parser = argparse.ArgumentParser(
description=self.get_description(),
prog=prog_name,
formatter_class=argparse.RawDescriptionHelpFormatter
)
return parser
slapos/cli/config.py 0 → 100644
View file @ 5f0924ff
# -*- coding: utf-8 -*-
import ConfigParser
import os
from slapos.cli.command import Command
class ConfigError(Exception):
pass
class ConfigCommand(Command):
"Base class for commands that require a configuration file"
log = None
default_config_var = 'SLAPOS_CONFIGURATION'
# use this if default_config_var does not exist
default_config_path = '/etc/opt/slapos/slapos.cfg'
def get_parser(self, prog_name):
ap = super(ConfigCommand, self).get_parser(prog_name)
ap.add_argument('--cfg', help='SlapOS configuration file - ' +
'defaults to $%s ' % self.default_config_var +
'or %s' % self.default_config_path)
return ap
def _get_config(self, cfg_path, required=False):
"""
Returns a configuration object if file exists/readable/valid,
None otherwise.
Will raise an error instead of returning None if required is True.
Even if required is False, may still raise an exception from the
configparser if the configuration content is very broken.
We don't catch that exception as it will clearly show what is
wrong with the file.
"""
if not os.path.exists(cfg_path):
if required:
raise ConfigError('Configuration file does not exist: %s' % cfg_path)
else:
return None
configp = ConfigParser.SafeConfigParser()
if configp.read(cfg_path) != [cfg_path]:
# bad permission, etc.
if required:
raise ConfigError('Cannot parse configuration file: %s' % cfg_path)
else:
return None
return configp
def fetch_config(self, args):
if args.cfg:
cfg_path = args.cfg
else:
cfg_path = os.environ.get(self.default_config_var, self.default_config_path)
cfg_path = os.path.expanduser(cfg_path)
self.log.debug('Loading config: %s' % cfg_path)
return self._get_config(cfg_path, required=True)
class ClientConfigCommand(ConfigCommand):
default_config_var = 'SLAPOS_CLIENT_CONFIGURATION'
default_config_path = '~/.slapos/slapos-client.cfg'
slapos/cli/console.py 0 → 100644
View file @ 5f0924ff
# -*- coding: utf-8 -*-
import logging
from slapos.cli.config import ClientConfigCommand
from slapos.client import init, do_console, ClientConfig
class ConsoleCommand(ClientConfigCommand):
"""
python console with slap library imported
You can play with the global "slap" object and
with the global "request" method.
examples :
>>> # Request instance
>>> request(kvm, "myuniquekvm")
>>> # Request software installation on owned computer
>>> supply(kvm, "mycomputer")
>>> # Fetch instance informations on already launched instance
>>> request(kvm, "myuniquekvm").getConnectionParameter("url")
"""
log = logging.getLogger('console')
def get_parser(self, prog_name):
ap = super(ConsoleCommand, self).get_parser(prog_name)
ap.add_argument('-u', '--master_url',
help='Url of SlapOS Master to use')
ap.add_argument('-k', '--key_file',
help='SSL Authorisation key file')
ap.add_argument('-c', '--cert_file',
help='SSL Authorisation certificate file')
return ap
def take_action(self, args):
configp = self.fetch_config(args)
conf = ClientConfig(args, configp)
local = init(conf)
do_console(local)
slapos/cli/entry.py 0 → 100644
View file @ 5f0924ff
# -*- coding: utf-8 -*-
import logging
import sys
import cliff
import cliff.app
from cliff.app import LOG
import cliff.commandmanager
import slapos.version
class SlapOSCommandManager(cliff.commandmanager.CommandManager):
def find_command(self, argv):
"""Given an argument list, find a command and
return the processor and any remaining arguments.
"""
# a little cheating, 'slapos node' is not documented by the help command
if argv == ['node']:
argv = ['node', 'status']
search_args = argv[:]
name = ''
while search_args:
if search_args[0].startswith('-'):
raise ValueError('Invalid command %r' % search_args[0])
next_val = search_args.pop(0)
name = '%s %s' % (name, next_val) if name else next_val
if name in self.commands:
cmd_ep = self.commands[name]
cmd_factory = cmd_ep.load()
return (cmd_factory, name, search_args)
else:
print >>sys.stderr, ('The command %r does not exist or is not yet implemented.\n'
'Please have a look at http://community.slapos.org to read documentation or forum.\n'
'Please also make sure that SlapOS Node is up to date.' % (argv,))
sys.exit(5)
class SlapOSApp(cliff.app.App):
#
# self.options.verbose_level:
# -q -> 0 (WARNING)
# -> 1 (INFO)
# -v -> 2 (DEBUG)
# -vv -> 3 (...)
# etc.
#
log = logging.getLogger('slapos')
CONSOLE_MESSAGE_FORMAT = '%(message)s'
LOG_FILE_MESSAGE_FORMAT = '[%(asctime)s] %(levelname)-8s %(name)s %(message)s'
def __init__(self):
super(SlapOSApp, self).__init__(
description='SlapOS client %s' % slapos.version.version,
version=slapos.version.version,
command_manager=SlapOSCommandManager('slapos.cli'),
)
def build_option_parser(self, *args, **kw):
kw.setdefault('argparse_kwargs', {})
kw['argparse_kwargs']['conflict_handler'] = 'resolve'
parser = super(SlapOSApp, self).build_option_parser(*args, **kw)
# add two aliases for --log-file (for compatibility with old commands)
parser.add_argument(
'--log-file', '--logfile', '--log_file',
action='store',
default=None,
help='Specify a file to log output. Only console by default.',
)
# always show tracebacks on errors
parser.set_defaults(debug=True)
return parser
def initialize_app(self, argv):
if self.options.verbose_level > 2:
self.log.debug('initialize_app')
def prepare_to_run_command(self, cmd):
if self.options.verbose_level > 2:
self.log.debug('prepare_to_run_command %s', cmd.__class__.__name__)
def clean_up(self, cmd, result, err):
if self.options.verbose_level > 2:
self.log.debug('clean_up %s', cmd.__class__.__name__)
if err:
self.log.debug('got an error: %s', err)
def run(self, argv):
# same as cliff.App.run except that it won't re-raise
# a logged exception
try:
self.options, remainder = self.parser.parse_known_args(argv)
self.configure_logging()
self.interactive_mode = not remainder
self.initialize_app(remainder)
except Exception as err:
if hasattr(self, 'options'):
debug = self.options.debug
else:
debug = True
if debug:
LOG.exception(err)
# XXX change from cliff behaviour: avoid
# displaying the exception twice
# raise
else:
LOG.error(err)
return 1
result = 1
if self.interactive_mode:
result = self.interact()
else:
result = self.run_subcommand(remainder)
return result
def main(argv=sys.argv[1:]):
app = SlapOSApp()
if not argv:
argv = ['-h']
return app.run(argv)
if __name__ == '__main__':
sys.exit(main(sys.argv[1:]))
slapos/cli/format.py 0 → 100644
View file @ 5f0924ff
# -*- coding: utf-8 -*-
import logging
import sys
from slapos.cli.config import ConfigCommand
from slapos.format import do_format, FormatConfig, tracing_monkeypatch, UsageError
class FormatCommand(ConfigCommand):
"""
create users, partitions and network configuration
"""
log = logging.getLogger('format')
def get_parser(self, prog_name):
ap = super(FormatCommand, self).get_parser(prog_name)
ap.add_argument('-x', '--computer_xml',
help="Path to file with computer's XML. If does not exists, will be created",
default=None)
ap.add_argument('--computer_json',
help="Path to a JSON version of the computer's XML (for development only).",
default=None)
ap.add_argument('-i', '--input_definition_file',
help="Path to file to read definition of computer instead of "
"declaration. Using definition file allows to disable "
"'discovery' of machine services and allows to define computer "
"configuration in fully controlled manner.")
ap.add_argument('-o', '--output_definition_file',
help="Path to file to write definition of computer from "
"declaration.")
ap.add_argument('-n', '--dry_run',
help="Don't actually do anything.",
default=False,
action="store_true")
ap.add_argument('--alter_user',
choices=['True', 'False'],
help="Shall slapformat alter user database [default: True]")
ap.add_argument('--alter_network',
choices=['True', 'False'],
help="Shall slapformat alter network configuration [default: True]")
ap.add_argument('--now',
help="Launch slapformat without delay",
default=False,
action="store_true")
return ap
def take_action(self, args):
configp = self.fetch_config(args)
conf = FormatConfig(logger=self.log)
conf.mergeConfig(args, configp)
if not self.app.options.log_file and conf.log_file:
# no log file is provided by argparser,
# we set up the one from config
file_handler = logging.FileHandler(conf.log_file)
formatter = logging.Formatter(self.app.LOG_FILE_MESSAGE_FORMAT)
file_handler.setFormatter(formatter)
self.log.addHandler(file_handler)
try:
conf.setConfig()
except UsageError as err:
sys.stderr.write(err.message + '\n')
sys.stderr.write("For help use --help\n")
sys.exit(1)
tracing_monkeypatch(conf)
try:
do_format(conf=conf)
except:
self.log.exception('Uncaught exception:')
raise
slapos/proxy/query.py → slapos/cli/proxy_show.py
View file @ 5f0924ff
# -*- coding: utf-8 -*-
# vim: set et sts=4:
import collections
import ConfigParser
from optparse import OptionParser, Option
import sys
import hashlib
import logging
import lxml.etree
import prettytable
import sqlite3
from slapos.cli.config import ConfigCommand
from slapos.proxy import ProxyConfig
from slapos.proxy.db_version import DB_VERSION
class Parser(OptionParser):
"""
Parse all arguments.
"""
def __init__(self, usage=None, version=None):
"""
Initialize all options possibles.
class ProxyShowCommand(ConfigCommand):
"""
OptionParser.__init__(self, usage=usage, version=version,
option_list=[
Option("-u", "--database-uri",
type=str,
help="URI for sqlite database"),
Option('--show-instances',
help="View instance information",
default=False,
action="store_true"),
Option('--show-params',
help="View published parameters",
default=False,
action="store_true"),
Option('--show-network',
help="View network information",
default=False,
action="store_true"),
Option('--show-all',
help="View all information",
default=False,
action="store_true"),
])
def check_args(self):
display proxy instances and parameters
"""
Check arguments
"""
(options, args) = self.parse_args()
if len(args) < 1:
self.error("Incorrect number of arguments")
return options, args[0]
log = logging.getLogger('proxy')
class Config:
def setConfig(self, option_dict, configuration_file_path):
"""
Set options given by parameters.
"""
# Set options parameters
for option, value in option_dict.__dict__.items():
setattr(self, option, value)
def get_parser(self, prog_name):
ap = super(ProxyShowCommand, self).get_parser(prog_name)
ap.add_argument('-u', '--database-uri',
help='URI for sqlite database')
# Load configuration file
configuration_parser = ConfigParser.SafeConfigParser()
configuration_parser.read(configuration_file_path)
# Merges the arguments and configuration
for section in ("slapproxy", "slapos"):
configuration_dict = dict(configuration_parser.items(section))
for key in configuration_dict:
if not getattr(self, key, None):
setattr(self, key, configuration_dict[key])
ap.add_argument('--computers',
help='view computer information',
action='store_true')
if not self.database_uri:
raise ValueError('database-uri is required.')
ap.add_argument('--software',
help='view software releases',
action='store_true')
ap.add_argument('--partitions',
help='view partitions',
action='store_true')
ap.add_argument('--slaves',
help='view slave instances',
action='store_true')
ap.add_argument('--params',
help='view published parameters',
action='store_true')
ap.add_argument('--network',
help='view network settings',
action='store_true')
return ap
def take_action(self, args):
configp = self.fetch_config(args)
conf = ProxyConfig(logger=self.log)
conf.mergeConfig(args, configp)
conf.setConfig()
do_show(conf=conf)
tbl_computer = 'computer' + DB_VERSION
tbl_software = 'software' + DB_VERSION
tbl_partition = 'partition' + DB_VERSION
tbl_partition_network = 'partition_network' + DB_VERSION
tbl_slave = 'slave' + DB_VERSION
null_str = u"-"
def coalesce(*seq):
el = None
for el in seq:
if el is not None:
return el
return el
def print_table(qry, tablename, skip=None):
......@@ -92,50 +76,33 @@ def print_table(qry, tablename, skip=None):
skip = set()
columns = [c[0] for c in qry.description if c[0] not in skip]
rows = []
for row in qry.fetchall():
line = {}
for col in columns:
val = row[col]
if val is None:
val = null_str
line[col] = val.strip()
rows.append(line)
max_width = {col: len(col) for col in columns}
for row in rows:
for col in columns:
val = row[col]
max_width[col] = max(max_width[col], len(val) if val else 0)
rows.append([coalesce(row[col], '-') for col in columns])
hdr = [col.center(max_width[col]) for col in columns]
pt = prettytable.PrettyTable(columns)
# https://code.google.com/p/prettytable/wiki/Tutorial
print
for row in rows:
pt.add_row(row)
if rows:
print 'table %s:' % tablename,
else:
print 'table %s: empty' % tablename
return
if skip:
print 'skipping %s' % ', '.join(skip)
else:
print
else:
print 'table %s: empty' % tablename
return
print ' | '.join(hdr)
print '-+-'.join('-'*len(h) for h in hdr)
for row in rows:
cells = [row[col].ljust(max_width[col]) for col in columns]
print ' | '.join(cells)
print pt.get_string(border=True, padding_width=0, vrules=prettytable.NONE)
def print_params(conn):
cur = conn.cursor()
print
qry = cur.execute("SELECT reference, partition_reference, software_type, connection_xml FROM %s" % tbl_partition)
for row in qry.fetchall():
if not row['connection_xml']:
......@@ -150,18 +117,19 @@ def print_params(conn):
if text and name in ('ssh-key', 'ssh-public-key'):
text = text[:20] + '...' + text[-20:]
print ' %s = %s' % (name, text)
print
def print_computer_table(conn):
tbl_computer = 'computer' + DB_VERSION
cur = conn.cursor()
qry = cur.execute("SELECT * FROM %s" % tbl_computer)
print_table(qry, tbl_computer)
def print_software_table(conn):
tbl_software = 'software' + DB_VERSION
cur = conn.cursor()
qry = cur.execute("SELECT * FROM %s" % tbl_software)
qry = cur.execute("SELECT *, md5(url) as md5 FROM %s" % tbl_software)
print_table(qry, tbl_software)
......@@ -170,21 +138,16 @@ def print_partition_table(conn):
qry = cur.execute("SELECT * FROM %s WHERE slap_state<>'free'" % tbl_partition)
print_table(qry, tbl_partition, skip=['xml', 'connection_xml', 'slave_instance_list'])
def print_slave_table(conn):
tbl_slave = 'slave' + DB_VERSION
cur = conn.cursor()
qry = cur.execute("SELECT * FROM %s" % tbl_slave)
print_table(qry, tbl_slave, skip=['connection_xml'])
def print_tables(conn):
print_computer_table(conn)
print_software_table(conn)
print_partition_table(conn)
print_slave_table(conn)
def print_network(conn):
print
tbl_partition_network = 'partition_network' + DB_VERSION
cur = conn.cursor()
addr = collections.defaultdict(list)
qry = cur.execute("""
......@@ -202,45 +165,40 @@ def print_network(conn):
print '%s: %s' % (partition_reference, ', '.join(addresses))
def run(config):
conn = sqlite3.connect(config.database_uri)
def do_show(conf):
conn = sqlite3.connect(conf.database_uri)
conn.row_factory = sqlite3.Row
fn = []
if config.show_all or config.show_instances:
fn.append(print_tables)
if config.show_all or config.show_params:
fn.append(print_params)
if config.show_all or config.show_network:
fn.append(print_network)
if fn:
for f in fn:
f(conn)
else:
print 'usage: %s [ --show-params | --show-network | --show-instances | --show-all ]' % sys.argv[0]
conn.create_function('md5', 1, lambda s: hashlib.md5(s).hexdigest())
print_all = not any(
[
conf.computers,
conf.software,
conf.partitions,
conf.slaves,
conf.params,
conf.network,
]
)
def main():
"Run default configuration."
usage = "usage: %s [options] CONFIGURATION_FILE" % sys.argv[0]
try:
# Parse arguments
config = Config()
config.setConfig(*Parser(usage=usage).check_args())
run(config)
return_code = 0
except SystemExit, err:
# Catch exception raise by optparse
return_code = err
sys.exit(return_code)
if print_all or conf.computers:
print_computer_table(conn)
print
if print_all or conf.software:
print_software_table(conn)
print
if print_all or conf.partitions:
print_partition_table(conn)
print
if print_all or conf.slaves:
print_slave_table(conn)
print
if print_all or conf.params:
print_params(conn)
print
if print_all or conf.network:
print_network(conn)
print
slapos/cli/proxy_start.py 0 → 100644
View file @ 5f0924ff
# -*- coding: utf-8 -*-
import logging
from slapos.cli.config import ConfigCommand
from slapos.proxy import do_proxy, ProxyConfig
class ProxyStartCommand(ConfigCommand):
"""
minimalist, stand-alone SlapOS Master
"""
log = logging.getLogger('proxy')
def get_parser(self, prog_name):
ap = super(ProxyStartCommand, self).get_parser(prog_name)
ap.add_argument('-u', '--database-uri',
help='URI for sqlite database')
return ap
def take_action(self, args):
configp = self.fetch_config(args)
conf = ProxyConfig(logger=self.log)
conf.mergeConfig(args, configp)
if not self.app.options.log_file and hasattr(conf, 'log_file'):
# no log file is provided by argparser,
# we set up the one from config
file_handler = logging.FileHandler(conf.log_file)
formatter = logging.Formatter(self.app.LOG_FILE_MESSAGE_FORMAT)
file_handler.setFormatter(formatter)
self.log.addHandler(file_handler)
conf.setConfig()
do_proxy(conf=conf)
slapos/cli/register.py 0 → 100644
View file @ 5f0924ff
# -*- coding: utf-8 -*-
import logging
import sys
from slapos.cli.command import Command
from slapos.register.register import do_register, RegisterConfig
class RegisterCommand(Command):
"""
register a node in the SlapOS cloud
"""
log = logging.getLogger('register')
def get_parser(self, prog_name):
ap = super(RegisterCommand, self).get_parser(prog_name)
ap.add_argument('node_name',
help='Name of the node')
ap.add_argument('--interface-name',
help='Interface name to access internet',
default='eth0')
ap.add_argument('--master-url',
help='URL of SlapOS master',
default='https://slap.vifib.com')
ap.add_argument('--master-url-web',
help='URL of SlapOS Master webservice to register certificates',
default='https://www.slapos.org')
ap.add_argument('--partition-number',
help='Number of partition on computer',
default='10',
type=int)
ap.add_argument('--ipv4-local-network',
help='Base of ipv4 local network',
default='10.0.0.0/16')
ap.add_argument('--ipv6-interface',
help='Interface name to get ipv6',
default='')
ap.add_argument('--login',
help='User login on SlapOS Master webservice')
ap.add_argument('--password',
help='User password on SlapOs Master webservice')
ap.add_argument('-t', '--create-tap',
help='Will trigger creation of one virtual "tap" interface per '
'Partition and attach it to primary interface. Requires '
'primary interface to be a bridge. defaults to false. '
'Needed to host virtual machines.',
default=False,
action='store_true')
ap.add_argument('-n', '--dry-run',
help='Simulate the execution steps',
default=False,
action='store_true')
return ap
def take_action(self, args):
try:
conf = RegisterConfig(logger=self.log)
conf.setConfig(args)
return_code = do_register(conf)
except SystemExit as err:
return_code = err
sys.exit(return_code)
slapos/cli/remove.py 0 → 100644
View file @ 5f0924ff
# -*- coding: utf-8 -*-
import logging
from slapos.cli.config import ClientConfigCommand
from slapos.client import init, do_remove, ClientConfig
class RemoveCommand(ClientConfigCommand):
"""
remove a Software from a node
"""
log = logging.getLogger('remove')
def get_parser(self, prog_name):
ap = super(RemoveCommand, self).get_parser(prog_name)
ap.add_argument('software_url',
help='Your software url')
ap.add_argument('node',
help="Target node")
return ap
def take_action(self, args):
configp = self.fetch_config(args)
conf = ClientConfig(args, configp)
local = init(conf)
do_remove(args.software_url, args.node, local)
slapos/cli/request.py 0 → 100644
View file @ 5f0924ff
# -*- coding: utf-8 -*-
import logging
from slapos.cli.config import ClientConfigCommand
from slapos.client import init, do_request, ClientConfig
def parse_option_dict(options):
"""
Parse a list of option strings like foo=bar baz=qux and return a dictionary.
Will raise if keys are repeated.
"""
ret = {}
for option_pair in (options or []):
key, value = option_pair.split('=', 1)
if key in ret:
raise ValueError("Multiple values provided for the same key '%s'" % key)
ret[key] = value
return ret
class RequestCommand(ClientConfigCommand):
"""request an instance and get status and parameters of instance"""
log = logging.getLogger('request')
def get_parser(self, prog_name):
ap = super(RequestCommand, self).get_parser(prog_name)
ap.add_argument('reference',
help='Your instance reference')
ap.add_argument('software_url',
help='Your software url')
# XXX TODO can this be an MD5 hash?
# XXX TODO can we do a minimal check for correctness of this argument?
# the alternative is a silent failure for mistyped/obsolete/invalid URL
ap.add_argument('--node',
nargs='+',
help="Node request option 'option1=value1 option2=value2'")
ap.add_argument('--type',
help='Software type to be requested')
ap.add_argument('--state',
help='State of the requested instance')
ap.add_argument('--slave',
action='store_true',
help='Ask for a slave instance')
ap.add_argument('--parameters',
nargs='+',
help="Give your configuration 'option1=value1 option2=value2'")
return ap
def take_action(self, args):
args.node = parse_option_dict(args.node)
args.parameters = parse_option_dict(args.parameters)
configp = self.fetch_config(args)
conf = ClientConfig(args, configp)
local = init(conf)
do_request(conf, local)
slapos/cli/slapgrid.py 0 → 100644
View file @ 5f0924ff
# -*- coding: utf-8 -*-
import logging
from slapos.cli.config import ConfigCommand
from slapos.grid.utils import setRunning, setFinished
from slapos.grid.slapgrid import (merged_options, check_missing_parameters, check_missing_files,
random_delay, create_slapgrid_object)
class SlapgridCommand(ConfigCommand):
log = None
method_name = NotImplemented
default_pidfile = NotImplemented
def get_parser(self, prog_name):
ap = super(SlapgridCommand, self).get_parser(prog_name)
# TODO move more options to the instance, software and report subclasses
ap.add_argument('--instance-root',
help='The instance root directory location.')
ap.add_argument('--software-root',
help='The software_root directory location.')
ap.add_argument('--master-url',
help='The master server URL. Mandatory.')
ap.add_argument('--computer-id',
help='The computer id defined in the server.')
ap.add_argument('--supervisord-socket',
help='The socket supervisor will use.')
ap.add_argument('--supervisord-configuration-path',
help='The location where supervisord configuration will be stored.')
ap.add_argument('--buildout', default=None,
help='Location of buildout binary.')
ap.add_argument('--pidfile',
help='The location where pidfile will be created. '
'Can be provided by configuration file, or defaults '
'to %s' % self.default_pidfile)
ap.add_argument('--key_file',
help='SSL Authorisation key file.')
ap.add_argument('--cert_file',
help='SSL Authorisation certificate file.')
ap.add_argument('--signature_private_key_file',
help='Signature private key file.')
ap.add_argument('--master_ca_file',
help='Root certificate of SlapOS master key.')
ap.add_argument('--certificate_repository_path',
help='Path to directory where downloaded certificates would be stored.')
ap.add_argument('--maximum-periodicity', type=int, default=None,
help='Periodicity at which buildout should be run in instance.')
ap.add_argument('--promise-timeout', type=int, default=3,
help='Promise timeout in seconds.')
ap.add_argument('--now', action='store_true',
help='Launch slapgrid without delay. Default behavior.')
return ap
def take_action(self, args):
configp = self.fetch_config(args)
options = merged_options(args, configp)
check_missing_parameters(options)
check_missing_files(options)
random_delay(options, logger=self.log)
slapgrid_object = create_slapgrid_object(options, logger=self.log)
pidfile = options.get('pidfile') or self.default_pidfile
if pidfile:
setRunning(logger=self.log, pidfile=pidfile)
try:
return getattr(slapgrid_object, self.method_name)()
finally:
if pidfile:
setFinished(pidfile)
class SoftwareCommand(SlapgridCommand):
"""run software installation/deletion"""
log = logging.getLogger('software')
method_name = 'processSoftwareReleaseList'
default_pidfile = '/opt/slapos/slapgrid-sr.pid'
def get_parser(self, prog_name):
ap = super(SoftwareCommand, self).get_parser(prog_name)
only = ap.add_mutually_exclusive_group()
only.add_argument('--all', action='store_true',
help='Process all Software Releases, even if already installed.')
only.add_argument('--only-sr', '--only',
help='Force the update of a single software release (can be full URL or MD5 hash), '
'even if is already installed. This option will make all other '
'sofware releases be ignored.')
return ap
class InstanceCommand(SlapgridCommand):
"""run instance deployment"""
log = logging.getLogger('instance')
method_name = 'processComputerPartitionList'
default_pidfile = '/opt/slapos/slapgrid-cp.pid'
def get_parser(self, prog_name):
ap = super(InstanceCommand, self).get_parser(prog_name)
only = ap.add_mutually_exclusive_group()
only.add_argument('--all', action='store_true',
help='Process all Computer Partitions.')
only.add_argument('--only-cp', '--only',
help='Update a single or a list of computer partitions '
'(ie.:slappartX, slappartY), '
'this option will make all other computer partitions be ignored.')
return ap
class ReportCommand(SlapgridCommand):
"""run instance reports and garbage collection"""
log = logging.getLogger('report')
method_name = 'agregateAndSendUsage'
default_pidfile = '/opt/slapos/slapgrid-ur.pid'
slapos/cli/supervisorctl.py 0 → 100644
View file @ 5f0924ff
# -*- coding: utf-8 -*-
import argparse
import logging
import os
from slapos.cli.config import ConfigCommand
from slapos.grid.svcbackend import launchSupervisord
import supervisor.supervisorctl
class SupervisorctlCommand(ConfigCommand):
"""enter into supervisor console, for process management"""
log = logging.getLogger('supervisorctl')
def get_parser(self, prog_name):
ap = super(SupervisorctlCommand, self).get_parser(prog_name)
ap.add_argument('supervisor_args',
nargs=argparse.REMAINDER,
help='parameters passed to supervisorctl')
return ap
def take_action(self, args):
configp = self.fetch_config(args)
instance_root = configp.get('slapos', 'instance_root')
configuration_file = os.path.join(instance_root, 'etc', 'supervisord.conf')
launchSupervisord(socket=os.path.join(instance_root, 'supervisord.socket'),
configuration_file=configuration_file,
logger=self.log)
supervisor.supervisorctl.main(args=['-c', configuration_file] + args.supervisor_args)
class SupervisorctlAliasCommand(SupervisorctlCommand):
def take_action(self, args):
args.supervisor_args = [self.alias] + args.supervisor_args
super(SupervisorctlAliasCommand, self).take_action(args)
class SupervisorctlStatusCommand(SupervisorctlAliasCommand):
"""alias for 'node supervisorctl status'"""
alias = 'status'
class SupervisorctlStartCommand(SupervisorctlAliasCommand):
"""alias for 'node supervisorctl start'"""
alias = 'start'
class SupervisorctlStopCommand(SupervisorctlAliasCommand):
"""alias for 'node supervisorctl stop'"""
alias = 'stop'
class SupervisorctlRestartCommand(SupervisorctlAliasCommand):
"""alias for 'node supervisorctl restart'"""
alias = 'restart'
class SupervisorctlTailCommand(SupervisorctlAliasCommand):
"""alias for 'node supervisorctl tail'"""
alias = 'tail'
slapos/cli/supervisord.py 0 → 100644
View file @ 5f0924ff
# -*- coding: utf-8 -*-
import logging
import os
from slapos.cli.config import ConfigCommand
from slapos.grid.svcbackend import launchSupervisord
class SupervisordCommand(ConfigCommand):
"""launch, if not already running, supervisor daemon"""
log = logging.getLogger('supervisord')
def take_action(self, args):
configp = self.fetch_config(args)
instance_root = configp.get('slapos', 'instance_root')
launchSupervisord(socket=os.path.join(instance_root, 'supervisord.socket'),
configuration_file=os.path.join(instance_root, 'etc', 'supervisord.conf'),
logger=self.log)
slapos/cli/supply.py 0 → 100644
View file @ 5f0924ff
# -*- coding: utf-8 -*-
import logging
from slapos.cli.config import ClientConfigCommand
from slapos.client import init, do_supply, ClientConfig
class SupplyCommand(ClientConfigCommand):
"""
supply a Software to a node
"""
log = logging.getLogger('supply')
def get_parser(self, prog_name):
ap = super(SupplyCommand, self).get_parser(prog_name)
ap.add_argument('software_url',
help='Your software url')
ap.add_argument('node',
help="Target node")
return ap
def take_action(self, args):
configp = self.fetch_config(args)
conf = ClientConfig(args, configp)
local = init(conf)
do_supply(args.software_url, args.node, local)
slapos/cli_legacy/bang.py 0 → 100644
View file @ 5f0924ff
# -*- coding: utf-8 -*-
##############################################################################
#
# Copyright (c) 2011, 2012 Vifib SARL and Contributors.
# All Rights Reserved.
#
# WARNING: This program as such is intended to be used by professional
# programmers who take the whole responsibility of assessing all potential
# consequences resulting from its eventual inadequacies and bugs
# End users who are looking for a ready-to-use solution with commercial
# guarantees and support are strongly advised to contract a Free Software
# Service Company
#
# This program is Free Software; you can redistribute it and/or
# modify it under the terms of the GNU General Public License
# as published by the Free Software Foundation; either version 3
# of the License, or (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.
#
##############################################################################
import argparse
import ConfigParser
from slapos.bang import do_bang
def main(*args):
ap = argparse.ArgumentParser()
ap.add_argument('-m', '--message', default='', help='Message for bang.')
ap.add_argument('configuration_file', type=argparse.FileType(),
help='SlapOS configuration file.')
if args:
args = ap.parse_args(list(args))
else:
args = ap.parse_args()
configp = ConfigParser.SafeConfigParser()
configp.readfp(args.configuration_file)
do_bang(configp, args.message)
slapos/cli_legacy/cache.py 0 → 100644
View file @ 5f0924ff
# -*- coding: utf-8 -*-
import argparse
import ConfigParser
from slapos.cache import do_lookup
def cache_lookup():
ap = argparse.ArgumentParser()
ap.add_argument("configuration_file", help="SlapOS configuration file")
ap.add_argument("software_url", help="Your software url or MD5 hash")
args = ap.parse_args()
configp = ConfigParser.SafeConfigParser()
configp.read(args.configuration_file)
do_lookup(configp, args.software_url)
slapos/cli_legacy/console.py 0 → 100644
View file @ 5f0924ff
# -*- coding: utf-8 -*-
import argparse
import os
import textwrap
from slapos.client import ClientConfig, init, do_console
from slapos.cli_legacy.util import get_config_parser
def console():
description = textwrap.dedent("""\
slapconsole allows you interact with slap API. You can play with the global
"slap" object and with the global "request" method.
examples :
>>> # Request instance
>>> request(kvm, "myuniquekvm")
>>> # Request software installation on owned computer
>>> supply(kvm, "mycomputer")
>>> # Fetch instance informations on already launched instance
>>> request(kvm, "myuniquekvm").getConnectionParameter("url")""")
ap = argparse.ArgumentParser(description=description,
formatter_class=argparse.RawDescriptionHelpFormatter)
ap.add_argument('-u', '--master_url',
default=None,
help='Url of SlapOS Master to use.')
ap.add_argument('-k', '--key_file',
help="SSL Authorisation key file.")
ap.add_argument('-c', '--cert_file',
help="SSL Authorisation certificate file.")
ap.add_argument('configuration_file',
help='path to slapos.cfg')
args = ap.parse_args()
if not os.path.isfile(args.configuration_file):
ap.error("%s: Not found or not a regular file." % args.configuration_file)
configp = get_config_parser(args.configuration_file)
conf = ClientConfig(args, configp)
local = init(conf)
do_console(local)
slapos/entry.py → slapos/cli_legacy/entry.py
View file @ 5f0924ff
......@@ -30,19 +30,20 @@ import argparse
import ConfigParser
import os
import sys
from slapos.bang import main as bang
from slapos.client import slapconsole as console
from slapos.client import request as request
from slapos.client import remove as remove
from slapos.client import supply as supply
from slapos.format import main as format
from slapos.cache import cache_lookup
from slapos.grid.slapgrid import runComputerPartition as instance
from slapos.grid.slapgrid import runSoftwareRelease as software
from slapos.grid.slapgrid import runUsageReport as report
from slapos.grid.svcbackend import supervisord
from slapos.grid.svcbackend import supervisorctl
from slapos.register.register import main as register
from slapos.cli_legacy.bang import main as bang
from slapos.cli_legacy.console import console
from slapos.cli_legacy.request import request
from slapos.cli_legacy.remove import remove
from slapos.cli_legacy.supply import supply
from slapos.cli_legacy.format import main as format
from slapos.cli_legacy.cache import cache_lookup
from slapos.cli_legacy.slapgrid import runComputerPartition as instance
from slapos.cli_legacy.slapgrid import runSoftwareRelease as software
from slapos.cli_legacy.slapgrid import runUsageReport as report
from slapos.cli_legacy.svcbackend import supervisord
from slapos.cli_legacy.svcbackend import supervisorctl
from slapos.cli_legacy.register import main as register
from slapos.version import version
# Note: this whole file is a hack. We should better try dedicated library
......@@ -60,6 +61,7 @@ class EntryPointNotImplementedError(NotImplementedError):
def __init__(self, *args, **kw_args):
NotImplementedError.__init__(self, *args, **kw_args)
def checkSlaposCfg():
"""
Check if a slapos configuration file was given as a argument.
......@@ -71,12 +73,13 @@ def checkSlaposCfg():
for element in sys.argv:
if '.cfg' in element:
if os.path.exists(element):
configuration = ConfigParser.SafeConfigParser()
configuration.read(element)
if configuration.has_section('slapos'):
configp = ConfigParser.SafeConfigParser()
configp.read(element)
if configp.has_section('slapos'):
return True
return False
def checkOption(option):
"""
Check if a given option is already in call line
......@@ -88,11 +91,12 @@ def checkOption(option):
if key in element:
return True
sys.argv.append(key)
if len(option) > 1 :
if len(option) > 1:
sys.argv = sys.argv + option[1:]
return True
def call(fun, config=False, option=None):
def call(fun, config_path=False, option=None):
"""
Add missing options to sys.argv
Add config if asked and it is missing
......@@ -102,12 +106,13 @@ def call(fun, config=False, option=None):
option = []
for element in option:
checkOption(element)
if config:
if config_path:
if not checkSlaposCfg():
sys.argv = [sys.argv[0]] + [os.path.expanduser(config)] + sys.argv[1:]
sys.argv = [sys.argv[0]] + [os.path.expanduser(config_path)] + sys.argv[1:]
fun()
sys.exit(0)
def dispatch(command, is_node_command):
""" Dispatch to correct SlapOS module.
Here we could use introspection to get rid of the big "if" statements,
......@@ -124,34 +129,34 @@ def dispatch(command, is_node_command):
if command == 'register':
call(register)
elif command == 'software':
call(software, config=GLOBAL_SLAPOS_CONFIGURATION,
call(software, config_path=GLOBAL_SLAPOS_CONFIGURATION,
option=['--pidfile /opt/slapos/slapgrid-sr.pid'])
elif command == 'instance':
call(instance, config=GLOBAL_SLAPOS_CONFIGURATION,
call(instance, config_path=GLOBAL_SLAPOS_CONFIGURATION,
option=['--pidfile /opt/slapos/slapgrid-cp.pid'])
elif command == 'report':
call(report, config=GLOBAL_SLAPOS_CONFIGURATION,
call(report, config_path=GLOBAL_SLAPOS_CONFIGURATION,
option=['--pidfile /opt/slapos/slapgrid-ur.pid'])
elif command == 'bang':
call(bang, config=GLOBAL_SLAPOS_CONFIGURATION)
call(bang, config_path=GLOBAL_SLAPOS_CONFIGURATION)
elif command == 'format':
call(format, config=GLOBAL_SLAPOS_CONFIGURATION, option=['-c', '-v'])
call(format, config_path=GLOBAL_SLAPOS_CONFIGURATION, option=['-c', '-v'])
elif command == 'supervisord':
call(supervisord, config=GLOBAL_SLAPOS_CONFIGURATION)
call(supervisord, config_path=GLOBAL_SLAPOS_CONFIGURATION)
elif command == 'supervisorctl':
call(supervisorctl, config=GLOBAL_SLAPOS_CONFIGURATION)
call(supervisorctl, config_path=GLOBAL_SLAPOS_CONFIGURATION)
elif command in ['start', 'stop', 'restart', 'status', 'tail']:
# Again, too hackish
sys.argv[-2:-2] = [command]
call(supervisorctl, config=GLOBAL_SLAPOS_CONFIGURATION)
call(supervisorctl, config_path=GLOBAL_SLAPOS_CONFIGURATION)
else:
return False
elif command == 'request':
call(request, config=USER_SLAPOS_CONFIGURATION)
call(request, config_path=USER_SLAPOS_CONFIGURATION)
elif command == 'supply':
call(supply, config=USER_SLAPOS_CONFIGURATION)
call(supply, config_path=USER_SLAPOS_CONFIGURATION)
elif command == 'remove':
call(remove, config=USER_SLAPOS_CONFIGURATION)
call(remove, config_path=USER_SLAPOS_CONFIGURATION)
elif command == 'start':
raise EntryPointNotImplementedError(command)
elif command == 'stop':
......@@ -159,12 +164,13 @@ def dispatch(command, is_node_command):
elif command == 'destroy':
raise EntryPointNotImplementedError(command)
elif command == 'console':
call(console, config=USER_SLAPOS_CONFIGURATION)
call(console, config_path=USER_SLAPOS_CONFIGURATION)
elif command == 'cache-lookup':
call(cache_lookup, config=GLOBAL_SLAPOS_CONFIGURATION)
call(cache_lookup, config_path=GLOBAL_SLAPOS_CONFIGURATION)
else:
return False
def main():
"""
Main entry point of SlapOS Node. Used to dispatch commands to python
......@@ -206,19 +212,19 @@ Node subcommands usage:
# Parse arguments
# XXX remove the "positional arguments" from help message
parser = argparse.ArgumentParser(usage=usage)
parser.add_argument('command')
parser.add_argument('argument_list', nargs=argparse.REMAINDER)
ap = argparse.ArgumentParser(usage=usage)
ap.add_argument('command')
ap.add_argument('argument_list', nargs=argparse.REMAINDER)
namespace = parser.parse_args()
args = ap.parse_args()
# Set sys.argv for the sub-entry point that we will call
command_line = [namespace.command]
command_line.extend(namespace.argument_list)
command_line = [args.command]
command_line.extend(args.argument_list)
sys.argv = command_line
try:
if not dispatch(namespace.command, is_node):
parser.print_help()
if not dispatch(args.command, is_node):
ap.print_help()
sys.exit(1)
except EntryPointNotImplementedError, exception:
print ('The command %s does not exist or is not yet implemented. Please '
......@@ -226,4 +232,3 @@ Node subcommands usage:
'forum. Please also make sure that SlapOS Node is up to '
'date.' % exception)
sys.exit(1)
slapos/cli_legacy/format.py 0 → 100644
View file @ 5f0924ff
# -*- coding: utf-8 -*-
##############################################################################
#
# Copyright (c) 2010, 2011, 2012 Vifib SARL and Contributors.
# All Rights Reserved.
#
# WARNING: This program as such is intended to be used by professional
# programmers who take the whole responsibility of assessing all potential
# consequences resulting from its eventual inadequacies and bugs
# End users who are looking for a ready-to-use solution with commercial
# guarantees and support are strongly advised to contract a Free Software
# Service Company
#
# This program is Free Software; you can redistribute it and/or
# modify it under the terms of the GNU General Public License
# as published by the Free Software Foundation; either version 3
# of the License, or (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.
#
##############################################################################
import argparse
import ConfigParser
import logging
import sys
import os
from slapos.format import FormatConfig, UsageError, tracing_monkeypatch, do_format
def main(*args):
"Run default configuration."
ap = argparse.ArgumentParser()
ap.add_argument('-x', '--computer_xml',
help="Path to file with computer's XML. If does not exists, will be created",
default=None)
ap.add_argument('--computer_json',
help="Path to a JSON version of the computer's XML (for development only).",
default=None)
ap.add_argument('-l', '--log_file',
help="The path to the log file used by the script.")
ap.add_argument('-i', '--input_definition_file',
help="Path to file to read definition of computer instead of "
"declaration. Using definition file allows to disable "
"'discovery' of machine services and allows to define computer "
"configuration in fully controlled manner.")
ap.add_argument('-o', '--output_definition_file',
help="Path to file to write definition of computer from "
"declaration.")
ap.add_argument('-n', '--dry_run',
help="Don't actually do anything.",
default=False,
action="store_true")
ap.add_argument('-v', '--verbose',
default=False,
action="store_true",
help="Verbose output.")
# the console option is actually ignored and not used anymore.
ap.add_argument('-c', '--console',
default=False,
action="store_true",
help="Console output.")
ap.add_argument('--alter_user',
choices=['True', 'False'],
help="Shall slapformat alter user database [default: True]")
ap.add_argument('--alter_network',
choices=['True', 'False'],
help="Shall slapformat alter network configuration [default: True]")
ap.add_argument('--now',
help="Launch slapformat without delay",
default=False,
action="store_true")
ap.add_argument('configuration_file',
help='path to slapos.cfg')
if args:
args = ap.parse_args(list(args))
else:
args = ap.parse_args()
logger = logging.getLogger("slapformat")
logger.addHandler(logging.StreamHandler())
if args.verbose:
logger.setLevel(logging.DEBUG)
logger.debug("Verbose mode enabled.")
else:
logger.setLevel(logging.INFO)
conf = FormatConfig(logger=logger)
configp = ConfigParser.SafeConfigParser()
if configp.read(args.configuration_file) != [args.configuration_file]:
raise UsageError('Cannot find or parse configuration file: %s' % args.configuration_file)
conf.mergeConfig(args, configp)
if conf.log_file:
if not os.path.isdir(os.path.dirname(conf.log_file)):
# fallback to console only if directory for logs does not exists and
# continue to run
raise ValueError('Please create directory %r to store %r log file' % (
os.path.dirname(conf.log_file), conf.log_file))
else:
file_handler = logging.FileHandler(conf.log_file)
file_handler.setFormatter(logging.Formatter("%(asctime)s - "
"%(name)s - %(levelname)s - %(message)s"))
conf.logger.addHandler(file_handler)
conf.logger.info('Configured logging to file %r' % conf.log_file)
try:
conf.setConfig()
except UsageError as exc:
sys.stderr.write(exc.message + '\n')
sys.stderr.write("For help use --help\n")
sys.exit(1)
tracing_monkeypatch(conf)
try:
do_format(conf=conf)
except:
conf.logger.exception('Uncaught exception:')
raise
slapos/cli_legacy/proxy_start.py 0 → 100644
View file @ 5f0924ff
# -*- coding: utf-8 -*-
# vim: set et sts=2:
import argparse
import ConfigParser
import logging
import os
import sys
from slapos.proxy import ProxyConfig, do_proxy
class UsageError(Exception):
pass
def main():
ap = argparse.ArgumentParser()
ap.add_argument('-l', '--log_file',
help='The path to the log file used by the script.')
ap.add_argument('-v', '--verbose',
action='store_true',
help='Verbose output.')
# XXX not used anymore, deprecated
ap.add_argument('-c', '--console',
action='store_true',
help='Console output.')
ap.add_argument('-u', '--database-uri',
help='URI for sqlite database')
ap.add_argument('configuration_file',
help='path to slapos.cfg')
args = ap.parse_args()
logger = logging.getLogger('slapproxy')
logger.addHandler(logging.StreamHandler())
if args.verbose:
logger.setLevel(logging.DEBUG)
else:
logger.setLevel(logging.INFO)
conf = ProxyConfig(logger=logger)
configp = ConfigParser.SafeConfigParser()
if configp.read(args.configuration_file) != [args.configuration_file]:
raise UsageError('Cannot find or parse configuration file: %s' % args.configuration_file)
conf.mergeConfig(args, configp)
if conf.log_file:
if not os.path.isdir(os.path.dirname(conf.log_file)):
raise ValueError('Please create directory %r to store %r log file' % (
os.path.dirname(conf.log_file), conf.log_file))
file_handler = logging.FileHandler(conf.log_file)
file_handler.setFormatter(logging.Formatter("%(asctime)s - %(name)s - %(levelname)s - %(message)s"))
logger.addHandler(file_handler)
logger.info('Configured logging to file %r' % conf.log_file)
conf.setConfig()
try:
do_proxy(conf=conf)
return_code = 0
except SystemExit as err:
return_code = err
sys.exit(return_code)
slapos/cli_legacy/register.py 0 → 100644
View file @ 5f0924ff
# -*- coding: utf-8 -*-
# vim: set et sts=2:
##############################################################################
#
# Copyright (c) 2012 Vifib SARL and Contributors. All Rights Reserved.
#
# WARNING: This program as such is intended to be used by professional
# programmers who take the whole responsibility of assessing all potential
# consequences resulting from its eventual inadequacies and bugs
# End users who are looking for a ready-to-use solution with commercial
# guarantees and support are strongly advised to contract a Free Software
# Service Company
#
# This program is Free Software; you can redistribute it and/or
# modify it under the terms of the GNU General Public License
# as published by the Free Software Foundation; either version 3
# of the License, or (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.
#
##############################################################################
import argparse
import logging
import sys
from slapos.register.register import do_register, RegisterConfig
def main():
ap = argparse.ArgumentParser()
ap.add_argument('node_name',
help='Name of the node')
ap.add_argument('--interface-name',
help='Interface name to access internet',
default='eth0')
ap.add_argument('--master-url',
help='URL of SlapOS master',
default='https://slap.vifib.com')
ap.add_argument('--master-url-web',
help='URL of SlapOS Master webservice to register certificates',
default='https://www.slapos.org')
ap.add_argument('--partition-number',
help='Number of partition on computer',
default='10',
type=int)
ap.add_argument('--ipv4-local-network',
help='Base of ipv4 local network',
default='10.0.0.0/16')
ap.add_argument('--ipv6-interface',
help='Interface name to get ipv6',
default='')
ap.add_argument('--login',
help='User login on SlapOS Master webservice')
ap.add_argument('--password',
help='User password on SlapOs Master webservice')
ap.add_argument('-t', '--create-tap',
help='Will trigger creation of one virtual "tap" interface per '
'Partition and attach it to primary interface. Requires '
'primary interface to be a bridge. defaults to false. '
'Needed to host virtual machines.',
default=False,
action='store_true')
ap.add_argument('-n', '--dry-run',
help='Simulate the execution steps',
default=False,
action='store_true')
args = ap.parse_args()
if args.password and not args.login:
ap.error('Please enter your login with your password')
logger = logging.getLogger('Register')
handler = logging.StreamHandler()
logger.setLevel(logging.DEBUG)
handler.setLevel(logging.INFO)
handler.setFormatter(logging.Formatter('%(levelname)s - %(message)s'))
logger.addHandler(handler)
try:
conf = RegisterConfig(logger=logger)
conf.setConfig(args)
return_code = do_register(conf)
except SystemExit as exc:
return_code = exc
sys.exit(return_code)
slapos/cli_legacy/remove.py 0 → 100644
View file @ 5f0924ff
# -*- coding: utf-8 -*-
import argparse
from slapos.client import ClientConfig, init, do_remove
from slapos.cli_legacy.util import get_config_parser
def remove():
ap = argparse.ArgumentParser()
ap.add_argument('configuration_file',
help='SlapOS configuration file')
ap.add_argument('software_url',
help='Your software url')
ap.add_argument('node',
help='Target node')
args = ap.parse_args()
configp = get_config_parser(args.configuration_file)
conf = ClientConfig(args, configp)
local = init(conf)
do_remove(args.software_url, args.node, local)
slapos/cli_legacy/request.py 0 → 100644
View file @ 5f0924ff
# -*- coding: utf-8 -*-
import argparse
from slapos.client import ClientConfig, init, do_request
from slapos.cli_legacy.util import get_config_parser
def argToDict(element):
"""
convert a table of string 'key=value' to dict
"""
if element is not None:
element_dict = dict([arg.split('=') for arg in element])
return element_dict
def request():
"""Run when invoking slapos request. Request an instance."""
# Parse arguments and inititate needed parameters
# XXX-Cedric: move argument parsing to main entry point
ap = argparse.ArgumentParser()
ap.add_argument('configuration_file',
help='SlapOS configuration file.')
ap.add_argument('reference',
help='Your instance reference')
ap.add_argument('software_url',
help='Your software url')
ap.add_argument('--node',
nargs='*',
help='Node request option '
"'option1=value1 option2=value2'")
ap.add_argument('--type',
type=str,
help='Define software type to be requested')
ap.add_argument('--slave',
action='store_true', default=False,
help='Ask for a slave instance')
ap.add_argument('--configuration',
nargs='*',
help='Give your configuration '
"'option1=value1 option2=value2'")
args = ap.parse_args()
if args.configuration:
args.parameters = argToDict(args.configuration)
if args.node:
args.node = argToDict(args.node)
configp = get_config_parser(args.configuration_file)
conf = ClientConfig(args, configp)
local = init(conf)
do_request(conf, local)
slapos/cli_legacy/slapgrid.py 0 → 100644
View file @ 5f0924ff
# -*- coding: utf-8 -*-
# vim: set et sts=2:
import argparse
import ConfigParser
import logging
import sys
from slapos.grid.utils import setRunning, setFinished
from slapos.grid.slapgrid import (merged_options, check_missing_parameters,
check_missing_files, random_delay, create_slapgrid_object)
def parse_arguments(*argument_tuple):
"""Parse arguments and return options dictionary merged with the config file."""
ap = argparse.ArgumentParser()
ap.add_argument('--instance-root',
help='The instance root directory location.')
ap.add_argument('--software-root',
help='The software_root directory location.')
ap.add_argument('--master-url',
help='The master server URL. Mandatory.')
ap.add_argument('--computer-id',
help='The computer id defined in the server.')
ap.add_argument('--supervisord-socket',
help='The socket supervisor will use.')
ap.add_argument('--supervisord-configuration-path',
help='The location where supervisord configuration will be stored.')
ap.add_argument('--buildout', default=None,
help='Location of buildout binary.')
ap.add_argument('--pidfile',
help='The location where pidfile will be created.')
ap.add_argument('--logfile',
help='The location where slapgrid logfile will be created.')
ap.add_argument('--key_file',
help='SSL Authorisation key file.')
ap.add_argument('--cert_file',
help='SSL Authorisation certificate file.')
ap.add_argument('--signature_private_key_file',
help='Signature private key file.')
ap.add_argument('--master_ca_file',
help='Root certificate of SlapOS master key.')
ap.add_argument('--certificate_repository_path',
help='Path to directory where downloaded certificates would be stored.')
ap.add_argument('-v', '--verbose', action='store_true',
help='Be verbose.')
ap.add_argument('--maximum-periodicity', type=int, default=None,
help='Periodicity at which buildout should be run in instance.')
ap.add_argument('--promise-timeout', type=int, default=3,
help='Promise timeout in seconds.')
ap.add_argument('--now', action='store_true',
help='Launch slapgrid without delay. Default behavior.')
ap.add_argument('--all', action='store_true',
help='Launch slapgrid to process all Software Releases '
'and/or Computer Partitions.')
ap.add_argument('--only-sr',
help='Force the update of a single software release (use url hash), '
'even if is already installed. This option will make all other '
'sofware releases be ignored.')
ap.add_argument('--only-cp',
help='Update a single or a list of computer partitions '
'(ie.:slappartX, slappartY), '
'this option will make all other computer partitions be ignored.')
ap.add_argument('configuration_file', type=argparse.FileType(),
help='SlapOS configuration file.')
# Deprecated options
ap.add_argument('-c', '--console', action='store_true',
help="Deprecated, doesn't do anything.")
ap.add_argument('--develop', action='store_true',
help='Deprecated, same as --all.')
ap.add_argument('--only_sr',
help='Deprecated, same as --only-sr.')
ap.add_argument('--only_cp',
help='Deprecated, same as --only-cp.')
ap.add_argument('--maximal_delay',
help='Deprecated. Will only work from configuration file in the future.')
if not argument_tuple:
args = ap.parse_args()
else:
args = ap.parse_args(list(argument_tuple))
return args
def setup_logger(options):
logger = logging.getLogger(__name__)
if options.get('logfile'):
handler = logging.FileHandler(options['logfile'])
else:
handler = logging.StreamHandler()
if options['verbose']:
handler.setLevel(logging.DEBUG)
else:
handler.setLevel(logging.INFO)
formatter = logging.Formatter(fmt='%(asctime)s %(name)-18s: '
'%(levelname)-8s %(message)s',
datefmt='%Y-%m-%dT%H:%M:%S')
handler.setFormatter(formatter)
logger.addHandler(handler)
return logger
def parseArgumentTupleAndReturnSlapgridObject(*argument_tuple):
"""Returns a new instance of slapgrid.Slapgrid created with argument+config parameters.
Also returns the pidfile path, and configures logger.
"""
args = parse_arguments(*argument_tuple)
configp = ConfigParser.SafeConfigParser()
configp.readfp(args.configuration_file)
options = merged_options(args, configp)
logger = setup_logger(options)
check_missing_parameters(options)
check_missing_files(options)
random_delay(options, logger=logger)
slapgrid_object = create_slapgrid_object(options, logger=logger)
return slapgrid_object, options.get('pidfile')
def realRun(argument_tuple, method):
slapgrid_object, pidfile = parseArgumentTupleAndReturnSlapgridObject(*argument_tuple)
if pidfile:
setRunning(logger=slapgrid_object.logger, pidfile=pidfile)
try:
return getattr(slapgrid_object, method)()
finally:
if pidfile:
setFinished(pidfile)
def runSoftwareRelease(*argument_tuple):
"""Hook for entry point to process Software Releases"""
sys.exit(realRun(argument_tuple, 'processSoftwareReleaseList'))
def runComputerPartition(*argument_tuple):
"""Hook for entry point to process Computer Partitions"""
sys.exit(realRun(argument_tuple, 'processComputerPartitionList'))
def runUsageReport(*argument_tuple):
"""Hook for entry point to process Usage Reports"""
sys.exit(realRun(argument_tuple, 'agregateAndSendUsage'))
slapos/cli_legacy/supply.py 0 → 100644
View file @ 5f0924ff
# -*- coding: utf-8 -*-
import argparse
from slapos.client import ClientConfig, init, do_supply
from slapos.cli_legacy.util import get_config_parser
def supply():
"""
Run when invoking slapos supply. Mostly argument parsing.
"""
ap = argparse.ArgumentParser()
ap.add_argument('configuration_file',
help='SlapOS configuration file')
ap.add_argument('software_url',
help='Your software url')
ap.add_argument('node',
help='Target node')
args = ap.parse_args()
configp = get_config_parser(args.configuration_file)
conf = ClientConfig(args, configp)
local = init(conf)
do_supply(args.software_url, args.node, local)
slapos/cli_legacy/svcbackend.py 0 → 100644
View file @ 5f0924ff
# -*- coding: utf-8 -*-
# vim: set et sts=2:
##############################################################################
#
# Copyright (c) 2010, 2011, 2012 Vifib SARL and Contributors.
# All Rights Reserved.
#
# WARNING: This program as such is intended to be used by professional
# programmers who take the whole responsibility of assessing all potential
# consequences resulting from its eventual inadequacies and bugs
# End users who are looking for a ready-to-use solution with commercial
# guarantees and support are strongly advised to contract a Free Software
# Service Company
#
# This program is Free Software; you can redistribute it and/or
# modify it under the terms of the GNU General Public License
# as published by the Free Software Foundation; either version 3
# of the License, or (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.
#
##############################################################################
import logging
import os
from optparse import OptionParser
import ConfigParser
from slapos.grid.svcbackend import launchSupervisord
def getOptionDict(*argument_tuple):
usage = """
Typical usage:
* %prog CONFIGURATION_FILE [arguments passed to supervisor]
""".strip()
parser = OptionParser(usage=usage)
# Parses arguments
if argument_tuple:
(argument_option_instance, argument_list) = parser.parse_args(list(argument_tuple))
else:
# No arguments given to entry point : we parse sys.argv.
(argument_option_instance, argument_list) = parser.parse_args()
if not argument_list:
parser.error("Configuration file is obligatory. Consult documentation by calling with -h.")
configuration_file = argument_list[0]
if not os.path.exists(configuration_file):
parser.error("Could not read configuration file : %s" % configuration_file)
slapgrid_configuration = ConfigParser.SafeConfigParser()
slapgrid_configuration.read(configuration_file)
# Merges the two dictionnaries
option_dict = dict(slapgrid_configuration.items("slapos"))
# Supervisord configuration location
option_dict.setdefault('supervisord_configuration_path',
os.path.join(option_dict['instance_root'], 'etc', 'supervisord.conf'))
# Supervisord socket
option_dict.setdefault('supervisord_socket',
os.path.join(option_dict['instance_root'], 'supervisord.socket'))
return option_dict, argument_list[1:]
def supervisorctl(*argument_tuple):
logger = logging.getLogger('SVCBackend')
logger.setLevel(logging.INFO)
handler = logging.StreamHandler()
logger.addHandler(handler)
option_dict, args = getOptionDict(*argument_tuple)
import supervisor.supervisorctl
launchSupervisord(socket=option_dict['supervisord_socket'],
configuration_file=option_dict['supervisord_configuration_path'],
logger=logger)
supervisor.supervisorctl.main(args=['-c', option_dict['supervisord_configuration_path']] + args)
def supervisord(*argument_tuple):
logger = logging.getLogger('SVCBackend')
logger.setLevel(logging.INFO)
handler = logging.StreamHandler()
logger.addHandler(handler)
option_dict, _ = getOptionDict(*argument_tuple)
launchSupervisord(socket=option_dict['supervisord_socket'],
configuration_file=option_dict['supervisord_configuration_path'],
logger=logger)
slapos/cli_legacy/util.py 0 → 100644
View file @ 5f0924ff
# -*- coding: utf-8 -*-
import ConfigParser
import os
def get_config_parser(path):
configp = ConfigParser.SafeConfigParser()
path = os.path.expanduser(path)
if not os.path.isfile(path):
raise OSError('Specified configuration file %s does not exist. Exiting.' % path)
configp.read(path)
return configp
slapos/client.py
View file @ 5f0924ff
......@@ -27,121 +27,37 @@
#
##############################################################################
import argparse
import atexit
import ConfigParser
import pprint
from optparse import OptionParser, Option
import os
from slapos.slap import ResourceNotReady
import slapos.slap.slap
import sys
import atexit
class Parser(OptionParser):
"""
Parse all arguments.
"""
def __init__(self, usage=None, version=None):
"""
Initialize all options possibles.
"""
OptionParser.__init__(self, usage=usage, version=version,
option_list=[
Option("-u", "--master_url",
default=None,
action="store",
help="Url of SlapOS Master to use."),
Option("-k", "--key_file",
action="store",
help="SSL Authorisation key file."),
Option("-c", "--cert_file",
action="store",
help="SSL Authorisation certificate file.")
])
def check_args(self):
"""
Check arguments
"""
(options, args) = self.parse_args()
if len(args) == 0:
self.error("Incorrect number of arguments")
elif not os.path.isfile(args[0]):
self.error("%s: Not found or not a regular file." % args[0])
# Return options and only first element of args since there is only one.
return options, args[0]
import pprint
def argToDict(element):
"""
convert a table of string 'key=value' to dict
"""
if element is not None:
element_dict = dict([arg.split('=') for arg in element])
return element_dict
import slapos.slap.slap
from slapos.slap import ResourceNotReady
def check_request_args():
"""
Parser for request
"""
parser = argparse.ArgumentParser()
parser.add_argument("configuration_file",
help="SlapOS configuration file.")
parser.add_argument("reference",
help="Your instance reference")
parser.add_argument("software_url",
help="Your software url")
parser.add_argument("--node",
nargs = '*',
help = "Node request option "
"'option1=value1 option2=value2'")
parser.add_argument("--type",
type = str,
help = "Define software type to be requested")
parser.add_argument("--slave",
action = "store_true", default=False,
help = "Ask for a slave instance")
parser.add_argument("--configuration",
nargs = '*',
help = "Give your configuration "
"'option1=value1 option2=value2'")
args = parser.parse_args()
# Convert to dict
if args.configuration is not None:
args.configuration = argToDict(args.configuration)
if args.node is not None:
args.node = argToDict(args.node)
return args
class ClientConfig(object):
state = None
class Config:
def __init__(self, option_dict, configuration_file_path=None):
def __init__(self, args, configp=None):
# XXX configp cannot possibly be optional
"""
Set options given by parameters.
"""
# Set options parameters
for option, value in option_dict.__dict__.items():
setattr(self, option, value)
for key, value in args.__dict__.items():
setattr(self, key, value)
# Load configuration file
configuration_parser = ConfigParser.SafeConfigParser()
if configuration_file_path:
configuration_file_path = os.path.expanduser(configuration_file_path)
if not os.path.isfile(configuration_file_path):
raise OSError('Specified configuration file %s does not exist.'
' Exiting.' % configuration_file_path)
configuration_parser.read(configuration_file_path)
# Merges the arguments and configuration
try:
configuration_dict = dict(configuration_parser.items('slapconsole'))
configuration_dict = dict(configp.items('slapconsole'))
except ConfigParser.NoSectionError:
pass
else:
for key in configuration_dict:
if not getattr(self, key, None):
setattr(self, key, configuration_dict[key])
configuration_dict = dict(configuration_parser.items('slapos'))
configuration_dict = dict(configp.items('slapos'))
master_url = configuration_dict.get('master_url', None)
# Backward compatibility, if no key and certificate given in option
# take one from slapos configuration
......@@ -158,67 +74,73 @@ class Config:
else:
setattr(self, 'master_url', master_url)
def init(config):
if self.key_file:
self.key_file = os.path.expanduser(self.key_file)
if self.cert_file:
self.cert_file = os.path.expanduser(self.cert_file)
def init(conf):
"""Initialize Slap instance, connect to server and create
aliases to common software releases"""
# XXX check certificate and key existence
slap = slapos.slap.slap()
slap.initializeConnection(config.master_url,
key_file=config.key_file, cert_file=config.cert_file)
slap.initializeConnection(conf.master_url,
key_file=conf.key_file, cert_file=conf.cert_file)
local = globals().copy()
local['slap'] = slap
# Create aliases as global variables
try:
alias = config.alias.split('\n')
alias = conf.alias.split('\n')
except AttributeError:
alias = []
software_list = []
for software in alias:
if software is not '':
if software:
name, url = software.split(' ')
software_list.append(name)
local[name] = url
# Create global variable too see available aliases
local['software_list'] = software_list
# Create global shortcut functions to request instance and software
def shorthandRequest(*args, **kwargs):
return slap.registerOpenOrder().request(*args, **kwargs)
def shorthandSupply(*args, **kwargs):
return slap.registerSupply().supply(*args, **kwargs)
local['request'] = shorthandRequest
local['supply'] = shorthandSupply
return local
def request():
"""Run when invoking slapos request. Request an instance."""
# Parse arguments and inititate needed parameters
# XXX-Cedric: move argument parsing to main entry point
options = check_request_args()
config = Config(options, options.configuration_file)
local = init(config)
# Request instance
print("Requesting %s..." % config.reference)
if config.software_url in local:
config.software_url = local[config.software_url]
def do_request(conf, local):
print("Requesting %s..." % conf.reference)
if conf.software_url in local:
conf.software_url = local[conf.software_url]
try:
partition = local['slap'].registerOpenOrder().request(
software_release = config.software_url,
partition_reference = config.reference,
partition_parameter_kw = config.configuration,
software_type = config.type,
filter_kw = config.node,
shared = config.slave
software_release = conf.software_url,
partition_reference = conf.reference,
partition_parameter_kw = conf.parameters,
software_type = conf.type,
filter_kw = conf.node,
state = conf.state,
shared = conf.slave
)
print "Instance requested.\nState is : %s." % partition.getState()
print "Connection parameters of instance are:"
pprint.pprint(partition.getConnectionParameterDict())
print "You can rerun command to get up-to-date informations."
except ResourceNotReady:
print("Instance requested. Master is provisionning it. Please rerun in a "
print("Instance requested. Master is provisioning it. Please rerun in a "
"couple of minutes to get connection informations.")
exit(2)
def _supply(software_url, computer_id, local, remove=False):
def do_supply(software_url, computer_id, local, remove=False):
"""
Request installation of Software Release
'software_url' on computer 'computer_id'.
......@@ -243,58 +165,12 @@ def _supply(software_url, computer_id, local, remove=False):
)
print 'Done.'
def supply():
"""
Run when invoking slapos supply. Mostly argument parsing.
"""
# XXX-Cedric: move argument parsing to main entry point
parser = argparse.ArgumentParser()
parser.add_argument("configuration_file",
help="SlapOS configuration file")
parser.add_argument("software_url",
help="Your software url")
parser.add_argument("node",
help="Target node")
args = parser.parse_args()
config = Config(args, args.configuration_file)
_supply(args.software_url, args.node, init(config))
def remove():
"""
Run when invoking slapos remove. Mostly argument parsing.
"""
# XXX-Cedric: move argument parsing to main entry point
parser = argparse.ArgumentParser()
parser.add_argument("configuration_file",
help="SlapOS configuration file.")
parser.add_argument("software_url",
help="Your software url")
parser.add_argument("node",
help="Target node")
args = parser.parse_args()
def do_remove(software_url, node, local):
do_supply(software_url, node, local, remove=True)
config = Config(args, args.configuration_file)
_supply(args.software_url, args.node, init(config), remove=True)
def slapconsole():
"""Ran when invoking slapconsole"""
# Parse arguments
usage = """usage: %s [options] CONFIGURATION_FILE
slapconsole allows you interact with slap API. You can play with the global
"slap" object and with the global "request" method.
examples :
>>> # Request instance
>>> request(kvm, "myuniquekvm")
>>> # Request software installation on owned computer
>>> supply(kvm, "mycomputer")
>>> # Fetch instance informations on already launched instance
>>> request(kvm, "myuniquekvm").getConnectionParameter("url")""" % sys.argv[0]
config = Config(*Parser(usage=usage).check_args())
local = init(config)
def do_console(local):
# try to enable readline with completion and history
try:
import readline
......@@ -316,4 +192,3 @@ examples :
atexit.register(save_history)
__import__("code").interact(banner="", local=local)
slapos/format.py
View file @ 5f0924ff
......@@ -28,8 +28,6 @@
#
##############################################################################
from optparse import OptionParser, Option
from xml_marshaller import xml_marshaller
import ConfigParser
import errno
import fcntl
......@@ -41,7 +39,6 @@ import netifaces
import os
import pwd
import random
import slapos.slap as slap
import shutil
import socket
import struct
......@@ -53,12 +50,10 @@ import traceback
import zipfile
import lxml.etree
from slapos.version import version
import xml_marshaller.xml_marshaller
# set up logging
logger = logging.getLogger("slapformat")
logger.setLevel(logging.INFO)
from slapos.util import mkdir_p
import slapos.slap as slap
def prettify_xml(xml):
......@@ -66,17 +61,14 @@ def prettify_xml(xml):
return lxml.etree.tostring(root, pretty_print=True)
from slapos.util import mkdir_p
class OS(object):
"""Wrap parts of the 'os' module to provide logging of performed actions."""
_os = os
def __init__(self, config):
self._dry_run = config.dry_run
self._verbose = config.verbose
self._logger = config.logger
def __init__(self, conf):
self._dry_run = conf.dry_run
self._logger = conf.logger
add = self._addWrapper
add('chown')
add('chmod')
......@@ -85,7 +77,6 @@ class OS(object):
def _addWrapper(self, name):
def wrapper(*args, **kw):
if self._verbose:
arg_list = [repr(x) for x in args] + [
'%s=%r' % (x, y) for x, y in kw.iteritems()
]
......@@ -161,36 +152,35 @@ def netmaskToPrefixIPv6(netmask):
netaddr.strategy.ipv6.str_to_int(netmask)]
def _getDict(instance):
def _getDict(obj):
"""
Serialize an object instance into dictionaries. List and dict will remains
Serialize an object into dictionaries. List and dict will remains
the same, basic type too. But encapsulated object will be returned as dict.
Set, collections and other aren't handle for now.
Args:
instance: an object of any type.
obj: an object of any type.
Returns:
A dictionary if the given object wasn't a list, a list otherwise.
"""
if isinstance(instance, list):
return [_getDict(item) for item in instance]
elif isinstance(instance, dict):
result = {}
for key in instance:
result[key] = _getDict(instance[key])
return result
if isinstance(obj, list):
return [_getDict(item) for item in obj]
if isinstance(obj, dict):
dikt = obj
else:
try:
dikt = instance.__dict__
dikt = obj.__dict__
except AttributeError:
return instance
result = {}
for key, value in dikt.iteritems():
result[key] = _getDict(value)
return result
return obj
return {
key: _getDict(value)
for key, value in dikt.iteritems()
# do not attempt to serialize logger: it is both useless and recursive.
if not isinstance(value, logging.Logger)
}
class Computer(object):
......@@ -248,30 +238,30 @@ class Computer(object):
# Can't find address
raise NoAddressOnInterface('No valid IPv6 found on %s.' % self.interface.name)
def send(self, config):
def send(self, conf):
"""
Send a marshalled dictionary of the computer object serialized via_getDict.
"""
slap_instance = slap.slap()
connection_dict = {}
if config.key_file and config.cert_file:
connection_dict['key_file'] = config.key_file
connection_dict['cert_file'] = config.cert_file
slap_instance.initializeConnection(config.master_url,
if conf.key_file and conf.cert_file:
connection_dict['key_file'] = conf.key_file
connection_dict['cert_file'] = conf.cert_file
slap_instance.initializeConnection(conf.master_url,
**connection_dict)
slap_computer = slap_instance.registerComputer(self.reference)
if config.dry_run:
if conf.dry_run:
return
try:
slap_computer.updateConfiguration(xml_marshaller.dumps(_getDict(self)))
slap_computer.updateConfiguration(xml_marshaller.xml_marshaller.dumps(_getDict(self)))
except slap.NotFoundError as error:
raise slap.NotFoundError("%s\nERROR : This SlapOS node is not recognised by "
"SlapOS Master. Please make sure computer_id of slapos.cfg looks "
"like 'COMP-123' and is correct.\nError is : 404 Not Found." % error)
def dump(self, path_to_xml, path_to_json):
def dump(self, path_to_xml, path_to_json, logger):
"""
Dump the computer object to an xml file via xml_marshaller.
......@@ -286,7 +276,7 @@ class Computer(object):
with open(path_to_json, 'wb') as fout:
fout.write(json.dumps(computer_dict, sort_keys=True, indent=2))
new_xml = xml_marshaller.dumps(computer_dict)
new_xml = xml_marshaller.xml_marshaller.dumps(computer_dict)
new_pretty_xml = prettify_xml(new_xml)
path_to_archive = path_to_xml + '.zip'
......@@ -340,7 +330,7 @@ class Computer(object):
A Computer object.
"""
dumped_dict = xml_marshaller.loads(open(path_to_xml).read())
dumped_dict = xml_marshaller.xml_marshaller.loads(open(path_to_xml).read())
# Reconstructing the computer object from the xml
computer = Computer(
......@@ -684,12 +674,13 @@ class Tap(object):
class Interface(object):
"""Represent a network interface on the system"""
def __init__(self, name, ipv4_local_network, ipv6_interface=None):
def __init__(self, logger, name, ipv4_local_network, ipv6_interface=None):
"""
Attributes:
name: String, the name of the interface
"""
self.logger = logger
self.name = str(name)
self.ipv4_local_network = ipv4_local_network
self.ipv6_interface = ipv6_interface
......@@ -853,7 +844,7 @@ class Interface(object):
if self._addSystemAddress(addr, netmask, False):
return dict(addr=addr, netmask=netmask)
else:
logger.warning('Impossible to add old local IPv4 %s. Generating '
self.logger.warning('Impossible to add old local IPv4 %s. Generating '
'new IPv4 address.' % addr)
return self._generateRandomIPv4Address(netmask)
else:
......@@ -911,7 +902,7 @@ class Interface(object):
# succeed, return it
return dict(addr=addr, netmask=netmask)
else:
logger.warning('Impossible to add old public IPv6 %s. '
self.logger.warning('Impossible to add old public IPv6 %s. '
'Generating new IPv6 address.' % addr)
# Try 10 times to add address, raise in case if not possible
......@@ -931,76 +922,8 @@ class Interface(object):
raise AddressGenerationError(addr)
class Parser(OptionParser):
"""
Parse all arguments.
"""
def __init__(self, usage=None, version=version):
"""
Initialize all options possibles.
"""
OptionParser.__init__(self, usage=usage, version=version,
option_list=[
Option("-x", "--computer_xml",
help="Path to file with computer's XML. If does not exists, "
"will be created",
default=None,
type=str),
Option("--computer_json",
help="Path to a JSON version of the computer's XML (for development only).",
default=None,
type=str),
Option("-l", "--log_file",
help="The path to the log file used by the script.",
type=str),
Option("-i", "--input_definition_file",
help="Path to file to read definition of computer instead of "
"declaration. Using definition file allows to disable "
"'discovery' of machine services and allows to define computer "
"configuration in fully controlled manner.",
type=str),
Option("-o", "--output_definition_file",
help="Path to file to write definition of computer from "
"declaration.",
type=str),
Option("-n", "--dry_run",
help="Don't actually do anything.",
default=False,
action="store_true"),
Option("-v", "--verbose",
default=False,
action="store_true",
help="Verbose output."),
Option("-c", "--console",
default=False,
action="store_true",
help="Console output."),
Option('--alter_user', choices=['True', 'False'],
help="Shall slapformat alter user database [default: True]"),
Option('--alter_network', choices=['True', 'False'],
help="Shall slapformat alter network configuration [default: True]"),
Option('--now',
help="Launch slapformat without delay",
default=False,
action="store_true"),
])
def check_args(self, args):
"""
Check arguments
"""
if args:
(options, args) = self.parse_args(list(args))
else:
(options, args) = self.parse_args()
if len(args) != 1:
self.error("Incorrect number of arguments")
return options, args[0]
def parse_computer_definition(config, definition_path):
config.logger.info('Using definition file %r' % definition_path)
def parse_computer_definition(conf, definition_path):
conf.logger.info('Using definition file %r' % definition_path)
computer_definition = ConfigParser.RawConfigParser({
'software_user': 'slapsoft',
})
......@@ -1010,20 +933,20 @@ def parse_computer_definition(config, definition_path):
netmask = None
if computer_definition.has_option('computer', 'address'):
address, netmask = computer_definition.get('computer', 'address').split('/')
if config.alter_network and config.interface_name is not None \
and config.ipv4_local_network is not None:
interface = Interface(config.interface_name, config.ipv4_local_network,
config.ipv6_interface)
if conf.alter_network and conf.interface_name is not None \
and conf.ipv4_local_network is not None:
interface = Interface(conf.logger, conf.interface_name, conf.ipv4_local_network,
conf.ipv6_interface)
computer = Computer(
reference=config.computer_id,
reference=conf.computer_id,
interface=interface,
addr=address,
netmask=netmask,
ipv6_interface=config.ipv6_interface,
ipv6_interface=conf.ipv6_interface,
software_user=computer_definition.get('computer', 'software_user'),
)
partition_list = []
for partition_number in range(int(config.partition_amount)):
for partition_number in range(int(conf.partition_amount)):
section = 'partition_%s' % partition_number
user = User(computer_definition.get(section, 'user'))
address_list = []
......@@ -1032,7 +955,7 @@ def parse_computer_definition(config, definition_path):
address_list.append(dict(addr=address, netmask=netmask))
tap = Tap(computer_definition.get(section, 'network_interface'))
partition = Partition(reference=computer_definition.get(section, 'pathname'),
path=os.path.join(config.instance_root,
path=os.path.join(conf.instance_root,
computer_definition.get(section, 'pathname')),
user=user,
address_list=address_list,
......@@ -1042,48 +965,48 @@ def parse_computer_definition(config, definition_path):
return computer
def parse_computer_xml(config, xml_path):
def parse_computer_xml(conf, xml_path):
if os.path.exists(xml_path):
config.logger.info('Loading previous computer data from %r' % xml_path)
conf.logger.info('Loading previous computer data from %r' % xml_path)
computer = Computer.load(xml_path,
reference=config.computer_id,
ipv6_interface=config.ipv6_interface)
reference=conf.computer_id,
ipv6_interface=conf.ipv6_interface)
# Connect to the interface defined by the configuration
computer.interface = Interface(config.interface_name, config.ipv4_local_network,
config.ipv6_interface)
computer.interface = Interface(conf.logger, conf.interface_name, conf.ipv4_local_network,
conf.ipv6_interface)
else:
# If no pre-existent configuration found, create a new computer object
config.logger.warning('Creating new data computer with id %r' % config.computer_id)
conf.logger.warning('Creating new data computer with id %r' % conf.computer_id)
computer = Computer(
reference=config.computer_id,
interface=Interface(config.interface_name, config.ipv4_local_network,
config.ipv6_interface),
reference=conf.computer_id,
interface=Interface(conf.logger, conf.interface_name, conf.ipv4_local_network,
conf.ipv6_interface),
addr=None,
netmask=None,
ipv6_interface=config.ipv6_interface,
software_user=config.software_user,
ipv6_interface=conf.ipv6_interface,
software_user=conf.software_user,
)
partition_amount = int(config.partition_amount)
partition_amount = int(conf.partition_amount)
existing_partition_amount = len(computer.partition_list)
if existing_partition_amount > partition_amount:
raise ValueError('Requested amount of computer partitions (%s) is lower '
'then already configured (%s), cannot continue' % (partition_amount,
len(computer.partition_list)))
config.logger.info('Adding %s new partitions' %
(partition_amount-existing_partition_amount))
conf.logger.info('Adding %s new partitions' %
(partition_amount - existing_partition_amount))
for nb_iter in range(existing_partition_amount, partition_amount):
# add new ones
user = User("%s%s" % (config.user_base_name, nb_iter))
user = User("%s%s" % (conf.user_base_name, nb_iter))
tap = Tap("%s%s" % (config.tap_base_name, nb_iter))
tap = Tap("%s%s" % (conf.tap_base_name, nb_iter))
path = os.path.join(config.instance_root, "%s%s" % (
config.partition_base_name, nb_iter))
path = os.path.join(conf.instance_root, "%s%s" % (
conf.partition_base_name, nb_iter))
computer.partition_list.append(
Partition(
reference="%s%s" % (config.partition_base_name, nb_iter),
reference="%s%s" % (conf.partition_base_name, nb_iter),
path=path,
user=user,
address_list=None,
......@@ -1093,7 +1016,7 @@ def parse_computer_xml(config, xml_path):
return computer
def write_computer_definition(config, computer):
def write_computer_definition(conf, computer):
computer_definition = ConfigParser.RawConfigParser()
computer_definition.add_section('computer')
if computer.address is not None and computer.netmask is not None:
......@@ -1109,44 +1032,58 @@ def write_computer_definition(config, computer):
computer_definition.set(section, 'user', partition.user.name)
computer_definition.set(section, 'network_interface', partition.tap.name)
computer_definition.set(section, 'pathname', partition.reference)
computer_definition.write(open(config.output_definition_file, 'w'))
config.logger.info('Stored computer definition in %r' % config.output_definition_file)
computer_definition.write(open(conf.output_definition_file, 'w'))
conf.logger.info('Stored computer definition in %r' % conf.output_definition_file)
def random_delay(conf):
# Add delay between 0 and 1 hour
# XXX should be the contrary: now by default, and cron should have
# --maximal-delay=3600
if not conf.now:
duration = float(60 * 60) * random.random()
print("Sleeping for %s seconds. To disable this feature, " \
"use with --now parameter in manual." % duration)
time.sleep(duration)
def run(config):
if config.input_definition_file:
computer = parse_computer_definition(config, config.input_definition_file)
def do_format(conf):
random_delay(conf)
if conf.input_definition_file:
computer = parse_computer_definition(conf, conf.input_definition_file)
else:
# no definition file, figure out computer
computer = parse_computer_xml(config, config.computer_xml)
computer = parse_computer_xml(conf, conf.computer_xml)
computer.instance_root = config.instance_root
computer.software_root = config.software_root
config.logger.info('Updating computer')
address = computer.getAddress(config.create_tap)
computer.instance_root = conf.instance_root
computer.software_root = conf.software_root
conf.logger.info('Updating computer')
address = computer.getAddress(conf.create_tap)
computer.address = address['addr']
computer.netmask = address['netmask']
if config.output_definition_file:
write_computer_definition(config, computer)
if conf.output_definition_file:
write_computer_definition(conf, computer)
computer.construct(alter_user=config.alter_user,
alter_network=config.alter_network,
create_tap=config.create_tap)
computer.construct(alter_user=conf.alter_user,
alter_network=conf.alter_network,
create_tap=conf.create_tap)
if getattr(config, 'certificate_repository_path', None):
mkdir_p(config.certificate_repository_path, mode=0o700)
if getattr(conf, 'certificate_repository_path', None):
mkdir_p(conf.certificate_repository_path, mode=0o700)
# Dumping and sending to the erp5 the current configuration
if not config.dry_run:
computer.dump(path_to_xml=config.computer_xml,
path_to_json=config.computer_json)
config.logger.info('Posting information to %r' % config.master_url)
computer.send(config)
config.logger.info('slapformat successfully prepared computer.')
if not conf.dry_run:
computer.dump(path_to_xml=conf.computer_xml,
path_to_json=conf.computer_json,
logger=conf.logger)
conf.logger.info('Posting information to %r' % conf.master_url)
computer.send(conf)
conf.logger.info('slapformat successfully prepared computer.')
class Config(object):
class FormatConfig(object):
key_file = None
cert_file = None
alter_network = None
......@@ -1155,14 +1092,14 @@ class Config(object):
computer_xml = None
computer_json = None
input_definition_file = None
logger = None
log_file = None
output_definition_file = None
verbose = None
dry_run = None
console = None
software_user = None
def __init__(self, logger):
self.logger = logger
@staticmethod
def checkRequiredBinary(binary_list):
missing_binary_list = []
......@@ -1179,33 +1116,26 @@ class Config(object):
raise UsageError('Some required binaries are missing or not '
'functional: %s' % (','.join(missing_binary_list), ))
def setConfig(self, option_dict, configuration_file_path):
def mergeConfig(self, args, configp):
"""
Set options given by parameters.
Must be executed before setting up the logger.
"""
self.key_file = None
self.cert_file = None
# set up logging
# XXX-Cedric: change code to use global logger
self.logger = logger
# Set options parameters
for option, value in option_dict.__dict__.items():
setattr(self, option, value)
# Load configuration file
configuration_parser = ConfigParser.SafeConfigParser()
if configuration_parser.read(configuration_file_path) != [configuration_file_path]:
raise UsageError('Cannot find or parse configuration file: %s' % configuration_file_path)
# Set argument parameters
for key, value in args.__dict__.items():
setattr(self, key, value)
# Merges the arguments and configuration
for section in ("slapformat", "slapos"):
configuration_dict = dict(configuration_parser.items(section))
configuration_dict = dict(configp.items(section))
for key in configuration_dict:
if not getattr(self, key, None):
setattr(self, key, configuration_dict[key])
def setConfig(self):
# setup some nones
for parameter in ['interface_name', 'partition_base_name', 'user_base_name',
'tap_base_name', 'ipv4_local_network', 'ipv6_interface']:
......@@ -1234,10 +1164,6 @@ class Config(object):
if self.create_tap is None:
self.create_tap = True
# Configure logging
if self.console:
self.logger.addHandler(logging.StreamHandler())
# Convert strings to booleans
for o in ['alter_network', 'alter_user', 'create_tap']:
attr = getattr(self, o)
......@@ -1275,22 +1201,9 @@ class Config(object):
if root_needed and os.getuid() != 0:
message = "Root rights are needed"
self.logger.error(message)
sys.stderr.write(message+'\n')
sys.stderr.write(message + '\n')
sys.exit()
if self.log_file:
if not os.path.isdir(os.path.dirname(self.log_file)):
# fallback to console only if directory for logs does not exists and
# continue to run
raise ValueError('Please create directory %r to store %r log file' % (
os.path.dirname(self.log_file), self.log_file))
else:
file_handler = logging.FileHandler(self.log_file)
file_handler.setFormatter(logging.Formatter("%(asctime)s - "
"%(name)s - %(levelname)s - %(message)s"))
self.logger.addHandler(file_handler)
self.logger.info('Configured logging to file %r' % self.log_file)
# Check mandatory options
for parameter in ('computer_id', 'instance_root', 'master_url',
'software_root', 'computer_xml'):
......@@ -1307,9 +1220,6 @@ class Config(object):
sys.exit(1)
self.logger.info("Started.")
if self.verbose:
self.logger.setLevel(logging.DEBUG)
self.logger.debug("Verbose mode enabled.")
if self.dry_run:
self.logger.info("Dry-run mode enabled.")
if self.create_tap:
......@@ -1325,22 +1235,22 @@ class Config(object):
self.output_definition_file = os.path.abspath(self.output_definition_file)
def tracing_monkeypatch(config):
def tracing_monkeypatch(conf):
"""Substitute os module and callAndRead function with tracing wrappers."""
global os
global callAndRead
real_callAndRead = callAndRead
os = OS(config)
if config.dry_run:
os = OS(conf)
if conf.dry_run:
def dry_callAndRead(argument_list, raise_on_error=True):
if argument_list == ['brctl', 'show']:
return real_callAndRead(argument_list, raise_on_error)
else:
return 0, ''
callAndRead = dry_callAndRead
def fake_getpwnam(user):
class result(object):
pw_uid = 12345
......@@ -1350,40 +1260,7 @@ def tracing_monkeypatch(config):
else:
dry_callAndRead = real_callAndRead
if config.verbose:
def logging_callAndRead(argument_list, raise_on_error=True):
config.logger.debug(' '.join(argument_list))
conf.logger.debug(' '.join(argument_list))
return dry_callAndRead(argument_list, raise_on_error)
callAndRead = logging_callAndRead
def main(*args):
"Run default configuration."
# Parse arguments
usage = "usage: %s [options] CONFIGURATION_FILE" % sys.argv[0]
options, configuration_file_path = Parser(usage=usage).check_args(args)
config = Config()
try:
config.setConfig(options, configuration_file_path)
except UsageError as err:
sys.stderr.write(err.message + '\n')
sys.stderr.write("For help use --help\n")
sys.exit(1)
tracing_monkeypatch(config)
# Add delay between 0 and 1 hour
# XXX should be the contrary: now by default, and cron should have
# --maximal-delay=3600
if not config.now:
duration = float(60*60) * random.random()
print("Sleeping for %s seconds. To disable this feature, " \
"use with --now parameter in manual." % duration)
time.sleep(duration)
try:
run(config)
except:
config.logger.exception('Uncaught exception:')
raise
slapos/grid/SlapObject.py
View file @ 5f0924ff
......@@ -28,50 +28,59 @@
#
##############################################################################
import logging
import os
import shutil
import subprocess
import pkg_resources
import pwd
import shutil
import stat
import subprocess
import tarfile
import tempfile
import textwrap
import utils
import xmlrpclib
from supervisor import xmlrpc
from slapos.grid.utils import (md5digest, getCleanEnvironment,
SlapPopen, dropPrivileges, updateFile)
from slapos.grid import utils # for mocked methods
from slapos.slap.slap import NotFoundError
from svcbackend import getSupervisorRPC
from exception import BuildoutFailedError, WrongPermissionError, \
PathDoesNotExistError
from networkcache import download_network_cached, upload_network_cached
from watchdog import getWatchdogID
from slapos.grid.svcbackend import getSupervisorRPC
from slapos.grid.exception import (BuildoutFailedError, WrongPermissionError,
PathDoesNotExistError)
from slapos.grid.networkcache import download_network_cached, upload_network_cached
from slapos.grid.watchdog import getWatchdogID
REQUIRED_COMPUTER_PARTITION_PERMISSION = '0750'
REQUIRED_COMPUTER_PARTITION_PERMISSION = 0o750
class Software(object):
"""This class is responsible for installing a software release"""
def __init__(self, url, software_root, buildout,
def __init__(self, url, software_root, buildout, logger,
signature_private_key_file=None, signature_certificate_list=None,
upload_cache_url=None, upload_dir_url=None, shacache_cert_file=None,
shacache_key_file=None, shadir_cert_file=None, shadir_key_file=None,
download_binary_cache_url=None, upload_binary_cache_url=None,
download_binary_dir_url=None, upload_binary_dir_url=None,
download_from_binary_cache_url_blacklist = [],
upload_to_binary_cache_url_blacklist = []):
download_from_binary_cache_url_blacklist=None,
upload_to_binary_cache_url_blacklist=None):
"""Initialisation of class parameters
"""
if download_from_binary_cache_url_blacklist is None:
download_from_binary_cache_url_blacklist = []
if upload_to_binary_cache_url_blacklist is None:
upload_to_binary_cache_url_blacklist = []
self.url = url
self.software_root = software_root
self.software_url_hash = utils.getSoftwareUrlHash(self.url)
self.software_url_hash = md5digest(self.url)
self.software_path = os.path.join(self.software_root,
self.software_url_hash)
self.buildout = buildout
self.logger = logging.getLogger('BuildoutManager')
self.logger = logger
self.signature_private_key_file = signature_private_key_file
self.signature_certificate_list = signature_certificate_list
self.upload_cache_url = upload_cache_url
......@@ -116,9 +125,8 @@ class Software(object):
else:
self._install_from_buildout()
# Upload to binary cache if possible and allowed
if (self.software_root and self.url and self.software_url_hash \
and self.upload_binary_cache_url \
and self.upload_binary_dir_url):
if all([self.software_root, self.url, self.software_url_hash,
self.upload_binary_cache_url, self.upload_binary_dir_url]):
blacklisted = False
for url in self.upload_to_binary_cache_url_blacklist:
if self.url.startswith(url):
......@@ -136,8 +144,8 @@ class Software(object):
it. If it fails, we notify the server.
"""
root_stat_info = os.stat(self.software_root)
os.environ = utils.getCleanEnvironment(pwd.getpwuid(root_stat_info.st_uid
).pw_dir)
os.environ = getCleanEnvironment(logger=self.logger,
home_path=pwd.getpwuid(root_stat_info.st_uid).pw_dir)
if not os.path.isdir(self.software_path):
os.mkdir(self.software_path)
extends_cache = tempfile.mkdtemp()
......@@ -145,40 +153,43 @@ class Software(object):
# In case when running as root copy ownership, to simplify logic
for path in [self.software_path, extends_cache]:
path_stat_info = os.stat(path)
if root_stat_info.st_uid != path_stat_info.st_uid or\
root_stat_info.st_gid != path_stat_info.st_gid:
if (root_stat_info.st_uid != path_stat_info.st_uid or
root_stat_info.st_gid != path_stat_info.st_gid):
os.chown(path, root_stat_info.st_uid,
root_stat_info.st_gid)
try:
buildout_parameter_list = [
'buildout:extends-cache=%s' % extends_cache,
'buildout:directory=%s' % self.software_path,]
'buildout:directory=%s' % self.software_path
]
if self.signature_private_key_file or \
self.upload_cache_url or \
self.upload_dir_url is not None:
if (self.signature_private_key_file or
self.upload_cache_url or
self.upload_dir_url):
buildout_parameter_list.append('buildout:networkcache-section=networkcache')
for buildout_option, value in (
for buildout_option, value in [
('%ssignature-private-key-file=%s', self.signature_private_key_file),
('%supload-cache-url=%s', self.upload_cache_url),
('%supload-dir-url=%s', self.upload_dir_url),
('%sshacache-cert-file=%s', self.shacache_cert_file),
('%sshacache-key-file=%s', self.shacache_key_file),
('%sshadir-cert-file=%s', self.shadir_cert_file),
('%sshadir-key-file=%s', self.shadir_key_file),
):
('%sshadir-key-file=%s', self.shadir_key_file)
]:
if value:
buildout_parameter_list.append( \
buildout_option % ('networkcache:', value))
buildout_parameter_list.append(buildout_option % ('networkcache:', value))
buildout_cfg = os.path.join(self.software_path, 'buildout.cfg')
self.createProfileIfMissing(buildout_cfg, self.url)
buildout_parameter_list.extend(['-c', buildout_cfg])
utils.bootstrapBuildout(self.software_path, self.buildout,
utils.bootstrapBuildout(path=self.software_path,
buildout=self.buildout,
logger=self.logger,
additional_buildout_parametr_list=buildout_parameter_list)
utils.launchBuildout(self.software_path,
os.path.join(self.software_path, 'bin', 'buildout'),
utils.launchBuildout(path=self.software_path,
buildout_binary=os.path.join(self.software_path, 'bin', 'buildout'),
logger=self.logger,
additional_buildout_parametr_list=buildout_parameter_list)
finally:
shutil.rmtree(extends_cache)
......@@ -226,7 +237,7 @@ class Software(object):
if func == os.path.islink:
os.unlink(path)
else:
os.chmod(path, 0600)
os.chmod(path, 0o600)
func(path)
try:
if os.path.exists(self.software_path):
......@@ -235,10 +246,8 @@ class Software(object):
else:
self.logger.info('Path %r does not exists, no need to remove.' %
self.software_path)
except IOError as error:
error_string = "I/O error while removing software (%s): %s" % (self.url,
error)
raise IOError(error_string)
except IOError as exc:
raise IOError("I/O error while removing software (%s): %s" % (self.url, exc))
class Partition(object):
......@@ -256,10 +265,12 @@ class Partition(object):
server_url,
software_release_url,
buildout,
logger,
certificate_repository_path=None,
):
"""Initialisation of class parameters"""
self.buildout = buildout
self.logger = logger
self.software_path = software_path
self.instance_path = instance_path
self.run_path = os.path.join(self.instance_path, 'etc', 'run')
......@@ -268,7 +279,6 @@ class Partition(object):
supervisord_partition_configuration_path
self.supervisord_socket = supervisord_socket
self.computer_partition = computer_partition
self.logger = logging.getLogger('Partition')
self.computer_id = computer_id
self.partition_id = partition_id
self.server_url = server_url
......@@ -284,19 +294,18 @@ class Partition(object):
self._updateCertificate()
def _updateCertificate(self):
if not os.path.exists(self.key_file) or \
not os.path.exists(self.cert_file):
if not os.path.exists(self.key_file) or not os.path.exists(self.cert_file):
self.logger.info('Certificate and key not found, downloading to %r and '
'%r' % (self.cert_file, self.key_file))
try:
partition_certificate = self.computer_partition.getCertificate()
except NotFoundError:
raise NotFoundError('Partition %s is not known from SlapOS Master.' % \
raise NotFoundError('Partition %s is not known from SlapOS Master.' %
self.partition_id)
open(self.key_file, 'w').write(partition_certificate['key'])
open(self.cert_file, 'w').write(partition_certificate['certificate'])
for f in [self.key_file, self.cert_file]:
os.chmod(f, 0400)
os.chmod(f, 0o400)
os.chown(f, *self.getUserGroupId())
def getUserGroupId(self):
......@@ -307,24 +316,24 @@ class Partition(object):
return (uid, gid)
def addServiceToGroup(self, partition_id,
runner_list, path, extension = ''):
runner_list, path, extension=''):
uid, gid = self.getUserGroupId()
program_partition_template = pkg_resources.resource_stream(__name__,
'templates/program_partition_supervisord.conf.in').read()
for runner in runner_list:
self.partition_supervisor_configuration += '\n' + \
program_partition_template % dict(
program_id='_'.join([partition_id, runner]),
program_directory=self.instance_path,
program_command=os.path.join(path, runner),
program_name=runner+extension,
instance_path=self.instance_path,
user_id=uid,
group_id=gid,
program_partition_template % {
'program_id': '_'.join([partition_id, runner]),
'program_directory': self.instance_path,
'program_command': os.path.join(path, runner),
'program_name': runner + extension,
'instance_path': self.instance_path,
'user_id': uid,
'group_id': gid,
# As supervisord has no environment to inherit, setup a minimalistic one
HOME=pwd.getpwuid(uid).pw_dir,
USER=pwd.getpwuid(uid).pw_name,
)
'HOME': pwd.getpwuid(uid).pw_dir,
'USER': pwd.getpwuid(uid).pw_name,
}
def updateSymlink(self, sr_symlink, software_path):
if os.path.lexists(sr_symlink):
......@@ -339,7 +348,7 @@ class Partition(object):
""" Creates configuration file from template in software_path, then
installs the software partition with the help of buildout
"""
self.logger.info("Installing Computer Partition %s..." \
self.logger.info("Installing Computer Partition %s..."
% self.computer_partition.getId())
# Checks existence and permissions of Partition directory
# Note : Partitions have to be created and configured before running slapgrid
......@@ -351,15 +360,14 @@ class Partition(object):
self.updateSymlink(sr_symlink, self.software_path)
instance_stat_info = os.stat(self.instance_path)
permission = oct(stat.S_IMODE(instance_stat_info.st_mode))
permission = stat.S_IMODE(instance_stat_info.st_mode)
if permission != REQUIRED_COMPUTER_PARTITION_PERMISSION:
raise WrongPermissionError('Wrong permissions in %s : actual ' \
'permissions are : %s, wanted ' \
'are %s' %
raise WrongPermissionError('Wrong permissions in %s: actual '
'permissions are: 0%o, wanted are 0%o' %
(self.instance_path, permission,
REQUIRED_COMPUTER_PARTITION_PERMISSION))
os.environ = utils.getCleanEnvironment(pwd.getpwuid(
instance_stat_info.st_uid).pw_dir)
os.environ = getCleanEnvironment(logger=self.logger,
home_path=pwd.getpwuid(instance_stat_info.st_uid).pw_dir)
# Generates buildout part from template
template_location = os.path.join(self.software_path, 'instance.cfg')
# Backward compatibility: "instance.cfg" file was named "template.cfg".
......@@ -369,23 +377,23 @@ class Partition(object):
self.logger.debug("Copying %r to %r" % (template_location, config_location))
try:
shutil.copy(template_location, config_location)
except IOError, e:
except IOError as exc:
# Template not found on SR, we notify user.
raise IOError('Software Release %s is not correctly installed.\n'
'%s' % (self.software_release_url, e))
raise IOError('Software Release %s is not correctly installed.\n%s' % (
self.software_release_url, exc))
# fill generated buildout with additional information
buildout_text = open(config_location).read()
buildout_text += '\n\n' + pkg_resources.resource_string(__name__,
'templates/buildout-tail.cfg.in') % dict(
computer_id=self.computer_id,
partition_id=self.partition_id,
server_url=self.server_url,
software_release_url=self.software_release_url,
key_file=self.key_file,
cert_file=self.cert_file
)
'templates/buildout-tail.cfg.in') % {
'computer_id': self.computer_id,
'partition_id': self.partition_id,
'server_url': self.server_url,
'software_release_url': self.software_release_url,
'key_file': self.key_file,
'cert_file': self.cert_file,
}
open(config_location, 'w').write(buildout_text)
os.chmod(config_location, 0640)
os.chmod(config_location, 0o640)
# Try to find the best possible buildout:
# *) if software_root/bin/bootstrap exists use this one to bootstrap
# locally
......@@ -418,12 +426,14 @@ class Partition(object):
self.logger.debug('Invoking %r in %r' % (' '.join(invocation_list),
self.instance_path))
process_handler = utils.SlapPopen(invocation_list,
preexec_fn=lambda: utils.dropPrivileges(uid, gid),
process_handler = SlapPopen(invocation_list,
preexec_fn=lambda: dropPrivileges(uid, gid, logger=self.logger),
cwd=self.instance_path,
env=utils.getCleanEnvironment(pwd.getpwuid(uid).pw_dir),
env=getCleanEnvironment(logger=self.logger,
home_path=pwd.getpwuid(uid).pw_dir),
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT)
stderr=subprocess.STDOUT,
logger=self.logger)
if process_handler.returncode is None or process_handler.returncode != 0:
message = 'Failed to bootstrap buildout in %r.' % (self.instance_path)
self.logger.error(message)
......@@ -432,18 +442,23 @@ class Partition(object):
if not os.path.exists(buildout_binary):
# use own buildout generation
utils.bootstrapBuildout(self.instance_path, self.buildout,
['buildout:bin-directory=%s'% os.path.join(self.instance_path,
'sbin')])
utils.bootstrapBuildout(path=self.instance_path,
buildout=self.buildout,
logger=self.logger,
additional_buildout_parameter_list=
['buildout:bin-directory=%s' %
os.path.join(self.instance_path, 'sbin')])
buildout_binary = os.path.join(self.instance_path, 'sbin', 'buildout')
# Launches buildout
utils.launchBuildout(self.instance_path, buildout_binary)
utils.launchBuildout(path=self.instance_path,
buildout_binary=buildout_binary,
logger=self.logger)
# Generates supervisord configuration file from template
self.logger.info("Generating supervisord config file from template...")
# check if CP/etc/run exists and it is a directory
# iterate over each file in CP/etc/run
# iterate over each file in CP/etc/service adding WatchdogID to their name
# if at least one is not 0750 raise -- partition has something funny
# if at least one is not 0o750 raise -- partition has something funny
runner_list = []
service_list = []
if os.path.exists(self.run_path):
......@@ -461,15 +476,16 @@ class Partition(object):
partition_id = self.computer_partition.getId()
group_partition_template = pkg_resources.resource_stream(__name__,
'templates/group_partition_supervisord.conf.in').read()
self.partition_supervisor_configuration = group_partition_template % dict(
instance_id=partition_id,
program_list=','.join(['_'.join([partition_id, runner])
for runner in runner_list+service_list]))
self.partition_supervisor_configuration = group_partition_template % {
'instance_id': partition_id,
'program_list': ','.join(['_'.join([partition_id, runner])
for runner in runner_list + service_list])
}
# Same method to add to service and run
self.addServiceToGroup(partition_id, runner_list,self.run_path)
self.addServiceToGroup(partition_id, service_list,self.service_path,
self.addServiceToGroup(partition_id, runner_list, self.run_path)
self.addServiceToGroup(partition_id, service_list, self.service_path,
extension=getWatchdogID())
utils.updateFile(self.supervisord_partition_configuration_path,
updateFile(self.supervisord_partition_configuration_path,
self.partition_supervisor_configuration)
self.updateSupervisor()
......@@ -481,21 +497,21 @@ class Partition(object):
partition_id = self.computer_partition.getId()
try:
supervisor.startProcessGroup(partition_id, False)
except xmlrpclib.Fault, e:
if e.faultString.startswith('BAD_NAME:'):
self.logger.info("Nothing to start on %s..." % \
except xmlrpclib.Fault as exc:
if exc.faultString.startswith('BAD_NAME:'):
self.logger.info("Nothing to start on %s..." %
self.computer_partition.getId())
else:
self.logger.info("Requested start of %s..." % self.computer_partition.getId())
def stop(self):
"""Asks supervisord to stop the instance."""
supervisor = self.getSupervisorRPC()
partition_id = self.computer_partition.getId()
try:
supervisor = self.getSupervisorRPC()
supervisor.stopProcessGroup(partition_id, False)
except xmlrpclib.Fault, e:
if e.faultString.startswith('BAD_NAME:'):
except xmlrpclib.Fault as exc:
if exc.faultString.startswith('BAD_NAME:'):
self.logger.info('Partition %s not known in supervisord, ignoring' % partition_id)
else:
self.logger.info("Requested stop of %s..." % self.computer_partition.getId())
......@@ -503,7 +519,7 @@ class Partition(object):
def destroy(self):
"""Destroys the partition and makes it available for subsequent use."
"""
self.logger.info("Destroying Computer Partition %s..." \
self.logger.info("Destroying Computer Partition %s..."
% self.computer_partition.getId())
# Launches "destroy" binary if exists
destroy_executable_location = os.path.join(self.instance_path, 'sbin',
......@@ -511,12 +527,14 @@ class Partition(object):
if os.path.exists(destroy_executable_location):
uid, gid = self.getUserGroupId()
self.logger.debug('Invoking %r' % destroy_executable_location)
process_handler = utils.SlapPopen([destroy_executable_location],
preexec_fn=lambda: utils.dropPrivileges(uid, gid),
process_handler = SlapPopen([destroy_executable_location],
preexec_fn=lambda: dropPrivileges(uid, gid, logger=self.logger),
cwd=self.instance_path,
env=utils.getCleanEnvironment(pwd.getpwuid(uid).pw_dir),
env=getCleanEnvironment(logger=self.logger,
home_path=pwd.getpwuid(uid).pw_dir),
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT)
stderr=subprocess.STDOUT,
logger=self.logger)
if process_handler.returncode is None or process_handler.returncode != 0:
message = 'Failed to destroy Computer Partition in %r.' % \
self.instance_path
......@@ -542,10 +560,8 @@ class Partition(object):
if os.path.exists(self.supervisord_partition_configuration_path):
os.remove(self.supervisord_partition_configuration_path)
self.updateSupervisor()
except IOError as error:
error_string = "I/O error while freeing partition (%s): %s" \
% (self.instance_path, error)
raise IOError(error_string)
except IOError as exc:
raise IOError("I/O error while freeing partition (%s): %s" % (self.instance_path, exc))
def fetchInformations(self):
"""Fetch usage informations with buildout, returns it.
......@@ -589,4 +605,3 @@ class Partition(object):
supervisor.addProcessGroup(gname)
self.logger.info('Updated %r' % gname)
self.logger.debug('Supervisord updated')
slapos/grid/slapgrid.py
View file @ 5f0924ff
......@@ -28,12 +28,6 @@
#
##############################################################################
import argparse
import ConfigParser
from exception import BuildoutFailedError
from hashlib import md5
from lxml import etree
import logging
import os
import pkg_resources
import random
......@@ -45,30 +39,22 @@ import tempfile
import time
import traceback
import warnings
if sys.version_info < (2, 6):
warnings.warn('Used python version (%s) is old and have problems with'
warnings.warn('Used python version (%s) is old and has problems with'
' IPv6 connections' % sys.version.split('\n')[0])
from lxml import etree
from slapos.slap.slap import NotFoundError
from slapos.slap.slap import ServerError
from SlapObject import Software, Partition, WrongPermissionError, \
PathDoesNotExistError
from svcbackend import launchSupervisord
from utils import createPrivateDirectory
from utils import dropPrivileges
from utils import getSoftwareUrlHash
from utils import setRunning
from utils import setFinished
from utils import SlapPopen
from utils import updateFile
from slapos import slap
MANDATORY_PARAMETER_LIST = [
'computer_id',
'instance_root',
'master_url',
'software_root',
]
from slapos.grid.exception import BuildoutFailedError
from slapos.grid.SlapObject import Software, Partition
from slapos.grid.svcbackend import launchSupervisord
from slapos.grid.utils import (md5digest, createPrivateDirectory, dropPrivileges,
SlapPopen, updateFile)
import slapos.slap
# XXX: should be moved to SLAP library
COMPUTER_PARTITION_DESTROYED_STATE = 'destroyed'
......@@ -90,310 +76,156 @@ class _formatXMLError(Exception):
pass
def parseArgumentTupleAndReturnSlapgridObject(*argument_tuple):
"""Parses arguments either from command line, from method parameters or from
config file. Then returns a new instance of slapgrid.Slapgrid with those
parameters. Also returns the options dict and unused variable list, and
configures logger.
"""
parser = argparse.ArgumentParser()
parser.add_argument("--instance-root",
help="The instance root directory location.")
parser.add_argument("--software-root",
help="The software_root directory location.")
parser.add_argument("--master-url",
help="The master server URL. Mandatory.")
parser.add_argument("--computer-id",
help="The computer id defined in the server.")
parser.add_argument("--supervisord-socket",
help="The socket supervisor will use.")
parser.add_argument("--supervisord-configuration-path",
help="The location where supervisord configuration will be stored.")
parser.add_argument("--buildout", default=None,
help="Location of buildout binary.")
parser.add_argument("--pidfile",
help="The location where pidfile will be created.")
parser.add_argument("--logfile",
help="The location where slapgrid logfile will be created.")
parser.add_argument("--key_file", help="SSL Authorisation key file.")
parser.add_argument("--cert_file",
help="SSL Authorisation certificate file.")
parser.add_argument("--signature_private_key_file",
help="Signature private key file.")
parser.add_argument("--master_ca_file",
help="Root certificate of SlapOS master key.")
parser.add_argument("--certificate_repository_path",
help="Path to directory where downloaded certificates would be stored.")
parser.add_argument("-c", "--console", action="store_true", default=False,
help="Deprecated, doesn't do anything.")
parser.add_argument("-v", "--verbose", action="store_true", default=False,
help="Be verbose.")
parser.add_argument("--maximum-periodicity", type=int, default=None,
help="Periodicity at which buildout should be run in instance.")
parser.add_argument("--promise-timeout", type=int, default=3,
help="Promise timeout in seconds.")
parser.add_argument("--now", action="store_true", default=False,
help="Launch slapgrid without delay. Default behavior.")
parser.add_argument("--all", action="store_true", default=False,
help="Launch slapgrid to process all Softare Releases "
"and/or Computer Partitions.")
parser.add_argument("--only-sr",
help="Force the update of a single software release (use url hash), "
"even if is already installed. This option will make all others "
"sofware releases be ignored.")
parser.add_argument("--only-cp",
help="Update a single or a list of computer partitions "
"(ie.:slappartX, slappartY),"
"this option will make all others computer partitions be ignored.")
parser.add_argument("configuration_file", nargs=1, type=argparse.FileType(),
help="SlapOS configuration file.")
# Deprecated options
parser.add_argument("--develop", action="store_true", default=False,
help="Deprecated, same as --all.")
parser.add_argument("--only_sr",
help="Deprecated, same as --only-sr.")
parser.add_argument("--only_cp",
help="Deprecated, same as --only-cp.")
parser.add_argument("--maximal_delay",
help="Deprecated. Will only work from configuration file in the future.")
def check_missing_parameters(options):
required = set([
'computer_id',
'instance_root',
'master_url',
'software_root',
])
if 'key_file' in options:
required.add('certificate_repository_path')
required.add('cert_file')
if 'cert_file' in options:
required.add('certificate_repository_path')
required.add('key_file')
missing = required.difference(options)
if missing:
raise RuntimeError('Missing mandatory parameters: %s' % ', '.join(sorted(missing)))
# Parses arguments
if argument_tuple == ():
# No arguments given to entry point : we parse sys.argv.
argument_option_instance = parser.parse_args()
else:
argument_option_instance = \
parser.parse_args(list(argument_tuple))
# Parses arguments from config file, if needed, then merge previous arguments
option_dict = {}
configuration_file = argument_option_instance.configuration_file[0]
# Loads config (if config specified)
slapgrid_configuration = ConfigParser.SafeConfigParser()
slapgrid_configuration.readfp(configuration_file)
# Merges the two dictionnaries
option_dict = dict(slapgrid_configuration.items("slapos"))
if slapgrid_configuration.has_section("networkcache"):
option_dict.update(dict(slapgrid_configuration.items("networkcache")))
for argument_key, argument_value in vars(argument_option_instance
).iteritems():
if argument_value is not None:
option_dict.update({argument_key: argument_value})
# Configures logger.
if option_dict['verbose']:
level = logging.DEBUG
else:
level = logging.INFO
logging.basicConfig(format='%(asctime)s %(levelname)-8s %(message)s',
level=level,
datefmt='%Y-%m-%dT%H:%M:%S')
if option_dict.get('logfile'):
console = logging.FileHandler(option_dict['logfile'])
console.setLevel(level)
console.setFormatter(logging.Formatter(
'%(asctime)s %(name)-18s: %(levelname)-8s %(message)s'))
logging.getLogger('').addHandler(console)
missing_mandatory_parameter_list = []
for mandatory_parameter in MANDATORY_PARAMETER_LIST:
if not mandatory_parameter in option_dict:
missing_mandatory_parameter_list.append(mandatory_parameter)
if option_dict.get('all') is True:
option_dict['develop'] = True
if option_dict.get('maximum_periodicity') is not None:
option_dict['force_periodicity'] = True
repository_required = False
if 'key_file' in option_dict:
repository_required = True
if not 'cert_file' in option_dict:
missing_mandatory_parameter_list.append('cert_file')
if 'cert_file' in option_dict:
repository_required = True
if not 'key_file' in option_dict:
missing_mandatory_parameter_list.append('key_file')
if repository_required:
if 'certificate_repository_path' not in option_dict:
missing_mandatory_parameter_list.append('certificate_repository_path')
if len(missing_mandatory_parameter_list) > 0:
parser.error('Missing mandatory parameters:\n%s' % '\n'.join(
missing_mandatory_parameter_list))
key_file = option_dict.get('key_file')
cert_file = option_dict.get('cert_file')
master_ca_file = option_dict.get('master_ca_file')
signature_private_key_file = option_dict.get('signature_private_key_file')
mandatory_file_list = [key_file, cert_file, master_ca_file]
# signature_private_key_file is not mandatory, we must be able to run
# slapgrid scripts without this parameter.
if signature_private_key_file:
mandatory_file_list.append(signature_private_key_file)
for k in ['shacache-cert-file', 'shacache-key-file', 'shadir-cert-file',
'shadir-key-file']:
mandatory_file_list.append(option_dict.get(k, None))
for f in mandatory_file_list:
if f is not None:
if not os.path.exists(f):
parser.error('File %r does not exist.' % f)
certificate_repository_path = option_dict.get('certificate_repository_path')
if certificate_repository_path is not None:
if not os.path.isdir(certificate_repository_path):
parser.error('Directory %r does not exist' %
certificate_repository_path)
def check_missing_files(options):
req_files = [
options.get('key_file'),
options.get('cert_file'),
options.get('master_ca_file'),
options.get('shacache-cert-file'),
options.get('shacache-key-file'),
options.get('shadir-cert-file'),
options.get('shadir-key-file'),
options.get('signature_private_key_file')
]
req_dirs = [
options.get('certificate_repository_path')
]
for f in req_files:
if f and not os.path.exists(f):
raise RuntimeError('File %r does not exist.' % f)
for d in req_dirs:
if d and not os.path.isdir(d):
raise RuntimeError('Directory %r does not exist' % d)
def merged_options(args, configp):
options = dict(configp.items('slapos'))
if configp.has_section('networkcache'):
options.update(dict(configp.items('networkcache')))
for key, value in vars(args).iteritems():
if value is not None:
options[key] = value
if options.get('all'):
options['develop'] = True
if options.get('maximum_periodicity') is not None:
options['force_periodicity'] = True
# Supervisord configuration location
if not option_dict.get('supervisord_configuration_path'):
option_dict['supervisord_configuration_path'] = \
os.path.join(option_dict['instance_root'], 'etc', 'supervisord.conf')
if not options.get('supervisord_configuration_path'):
options['supervisord_configuration_path'] = \
os.path.join(options['instance_root'], 'etc', 'supervisord.conf')
# Supervisord socket
if not option_dict.get('supervisord_socket'):
option_dict['supervisord_socket'] = \
os.path.join(option_dict['instance_root'], 'supervisord.socket')
signature_certificate_list_string = \
option_dict.get('signature-certificate-list', None)
if signature_certificate_list_string is not None:
cert_marker = "-----BEGIN CERTIFICATE-----"
signature_certificate_list = [cert_marker + '\n' + q.strip() \
for q in signature_certificate_list_string.split(cert_marker) \
if q.strip()]
else:
signature_certificate_list = None
if not options.get('supervisord_socket'):
options['supervisord_socket'] = \
os.path.join(options['instance_root'], 'supervisord.socket')
# Parse cache / binary cache options
# Backward compatibility about "binary-cache-url-blacklist" deprecated option
if option_dict.get("binary-cache-url-blacklist") and not \
option_dict.get("download-from-binary-cache-url-blacklist"):
option_dict["download-from-binary-cache-url-blacklist"] = \
option_dict["binary-cache-url-blacklist"]
option_dict["download-from-binary-cache-url-blacklist"] = [
url.strip() for url in option_dict.get(
if options.get("binary-cache-url-blacklist") and not \
options.get("download-from-binary-cache-url-blacklist"):
options["download-from-binary-cache-url-blacklist"] = \
options["binary-cache-url-blacklist"]
options["download-from-binary-cache-url-blacklist"] = [
url.strip() for url in options.get(
"download-from-binary-cache-url-blacklist", "").split('\n') if url]
option_dict["upload-to-binary-cache-url-blacklist"] = [
url.strip() for url in option_dict.get(
options["upload-to-binary-cache-url-blacklist"] = [
url.strip() for url in options.get(
"upload-to-binary-cache-url-blacklist", "").split('\n') if url]
# Sleep for a random time to avoid SlapOS Master being DDOSed by an army of
# SlapOS Nodes configured with cron.
if option_dict["now"]:
# XXX-Cedric: deprecate "--now"
maximal_delay = 0
else:
maximal_delay = int(option_dict.get("maximal_delay", "0"))
if maximal_delay > 0:
return options
def random_delay(options, logger):
"""
Sleep for a random time to avoid SlapOS Master being DDOSed by an army of
SlapOS Nodes configured with cron.
"""
if options['now']:
# XXX-Cedric: deprecate '--now'
return
maximal_delay = int(options.get('maximal_delay', '0'))
if maximal_delay:
duration = random.randint(1, maximal_delay)
logging.info("Sleeping for %s seconds. To disable this feature, " \
"check --now parameter in slapgrid help." % duration)
logger.info('Sleeping for %s seconds. To disable this feature, ' \
'check --now parameter in slapgrid help.' % duration)
time.sleep(duration)
# Return new Slapgrid instance and options
return ([Slapgrid(software_root=option_dict['software_root'],
instance_root=option_dict['instance_root'],
master_url=option_dict['master_url'],
computer_id=option_dict['computer_id'],
supervisord_socket=option_dict['supervisord_socket'],
supervisord_configuration_path=option_dict[
'supervisord_configuration_path'],
key_file=key_file,
cert_file=cert_file,
master_ca_file=master_ca_file,
certificate_repository_path=certificate_repository_path,
signature_private_key_file=signature_private_key_file,
def create_slapgrid_object(options, logger):
signature_certificate_list = None
if 'signature-certificate-list' in options:
cert_marker = '-----BEGIN CERTIFICATE-----'
signature_certificate_list = [
cert_marker + '\n' + q.strip()
for q in options['signature-certificate-list'].split(cert_marker)
if q.strip()
]
op = options
return Slapgrid(software_root=op['software_root'],
instance_root=op['instance_root'],
master_url=op['master_url'],
computer_id=op['computer_id'],
supervisord_socket=op['supervisord_socket'],
supervisord_configuration_path=op['supervisord_configuration_path'],
buildout=op.get('buildout'),
logger=logger,
force_periodicity = op.get('force_periodicity', False),
maximum_periodicity = op.get('maximum_periodicity', 86400),
key_file=op.get('key_file'),
cert_file=op.get('cert_file'),
signature_private_key_file=op.get('signature_private_key_file'),
signature_certificate_list=signature_certificate_list,
download_binary_cache_url=\
option_dict.get('download-binary-cache-url', None),
upload_binary_cache_url=\
option_dict.get('upload-binary-cache-url', None),
download_binary_cache_url=op.get('download-binary-cache-url'),
upload_binary_cache_url=op.get('upload-binary-cache-url'),
download_from_binary_cache_url_blacklist=\
option_dict.get('download-from-binary-cache-url-blacklist', []),
op.get('download-from-binary-cache-url-blacklist', []),
upload_to_binary_cache_url_blacklist=\
option_dict.get('upload-to-binary-cache-url-blacklist', []),
upload_cache_url=option_dict.get('upload-cache-url', None),
download_binary_dir_url=\
option_dict.get('download-binary-dir-url', None),
upload_binary_dir_url=\
option_dict.get('upload-binary-dir-url', None),
upload_dir_url=option_dict.get('upload-dir-url', None),
buildout=option_dict.get('buildout'),
promise_timeout=option_dict['promise_timeout'],
shacache_cert_file=option_dict.get('shacache-cert-file', None),
shacache_key_file=option_dict.get('shacache-key-file', None),
shadir_cert_file=option_dict.get('shadir-cert-file', None),
shadir_key_file=option_dict.get('shadir-key-file', None),
develop=option_dict.get('develop', False),
software_release_filter_list=option_dict.get('only-sr',
op.get('upload-to-binary-cache-url-blacklist', []),
upload_cache_url=op.get('upload-cache-url'),
download_binary_dir_url=op.get('download-binary-dir-url'),
upload_binary_dir_url=op.get('upload-binary-dir-url'),
upload_dir_url=op.get('upload-dir-url'),
master_ca_file=op.get('master_ca_file'),
certificate_repository_path=op.get('certificate_repository_path'),
promise_timeout=op['promise_timeout'],
shacache_cert_file=op.get('shacache-cert-file'),
shacache_key_file=op.get('shacache-key-file'),
shadir_cert_file=op.get('shadir-cert-file'),
shadir_key_file=op.get('shadir-key-file'),
develop=op.get('develop', False),
# Try to fetch from deprecated argument
option_dict.get('only_sr', None)),
computer_partition_filter_list=option_dict.get('only-cp',
software_release_filter_list=op.get('only-sr', op.get('only_sr')),
# Try to fetch from deprecated argument
option_dict.get('only_cp', None)),
force_periodicity = option_dict.get('force_periodicity', False),
maximum_periodicity = option_dict.get('maximum_periodicity', 86400),
),
option_dict])
def realRun(argument_tuple, method_list):
slapgrid_object, option_dict = \
parseArgumentTupleAndReturnSlapgridObject(*argument_tuple)
pidfile = option_dict.get('pidfile')
if pidfile:
setRunning(pidfile)
try:
failed = False
failed_promise = False
for method in method_list:
# Quite complicated way to figure out if everything went fine
return_value = getattr(slapgrid_object, method)()
if return_value == SLAPGRID_FAIL:
failed = True
if return_value == SLAPGRID_PROMISE_FAIL:
failed_promise = True
finally:
if pidfile:
setFinished(pidfile)
if failed:
sys.exit(SLAPGRID_FAIL)
if failed_promise:
sys.exit(SLAPGRID_PROMISE_FAIL)
sys.exit(SLAPGRID_SUCCESS)
def run(*argument_tuple):
"""Hooks for generic entry point to proces Software Releases (sr),
Computer Partitions (cp) and Usage Reports (ur)
Will run one by one each task (sr, cp, ur). If specified,
will run in the user wanted order.
"""
realRun(argument_tuple, ['processSoftwareReleaseList',
'processComputerPartitionList', 'agregateAndSendUsage'])
def runSoftwareRelease(*argument_tuple):
"""Hook for entry point to process Software Releases only
"""
realRun(argument_tuple, ['processSoftwareReleaseList'])
def runComputerPartition(*argument_tuple):
"""Hook for entry point to process Computer Partitions only
"""
realRun(argument_tuple, ['processComputerPartitionList'])
def runUsageReport(*argument_tuple):
"""Hook for entry point to process Usage Reports only
"""
realRun(argument_tuple, ['agregateAndSendUsage'])
computer_partition_filter_list=op.get('only-cp', op.get('only_cp')))
class Slapgrid(object):
......@@ -412,6 +244,7 @@ class Slapgrid(object):
supervisord_socket,
supervisord_configuration_path,
buildout,
logger,
force_periodicity=False,
maximum_periodicity=86400,
key_file=None,
......@@ -465,17 +298,15 @@ class Slapgrid(object):
self.shacache_key_file = shacache_key_file
self.shadir_cert_file = shadir_cert_file
self.shadir_key_file = shadir_key_file
# Configures logger
self.logger = logging.getLogger('Slapgrid')
self.logger = logger
# Creates objects from slap module
self.slap = slap.slap()
self.slap = slapos.slap.slap()
self.slap.initializeConnection(self.master_url, key_file=self.key_file,
cert_file=self.cert_file, master_ca_file=self.master_ca_file)
self.computer = self.slap.registerComputer(self.computer_id)
# Defines all needed paths
self.instance_etc_directory = os.path.join(self.instance_root, 'etc')
self.supervisord_configuration_directory = \
os.path.join(self.instance_etc_directory, 'supervisord.conf.d')
os.path.join(self.instance_root, 'etc', 'supervisord.conf.d')
self.buildout = buildout
self.promise_timeout = promise_timeout
self.develop = develop
......@@ -494,7 +325,7 @@ class Slapgrid(object):
def getWatchdogLine(self):
invocation_list = [WATCHDOG_PATH]
invocation_list.append("--master-url '%s' " % self.master_url)
if self.certificate_repository_path is not None:
if self.certificate_repository_path:
invocation_list.append("--certificate-repository-path '%s'" \
% self.certificate_repository_path)
invocation_list.append("--computer-id '%s'" % self.computer_id)
......@@ -506,64 +337,58 @@ class Slapgrid(object):
"""
# Checks for software_root and instance_root existence
if not os.path.isdir(self.software_root):
error = "%s does not exist." % self.software_root
raise OSError(error)
raise OSError('%s does not exist.' % self.software_root)
if not os.path.isdir(self.instance_root):
error = "%s does not exist." % self.instance_root
raise OSError(error)
raise OSError('%s does not exist.' % self.instance_root)
# Creates everything needed
try:
# Creates instance_root structure
createPrivateDirectory(self.instance_etc_directory)
createPrivateDirectory(os.path.join(self.instance_root, 'var'))
createPrivateDirectory(os.path.join(self.instance_root, 'var', 'log'))
createPrivateDirectory(os.path.join(self.instance_root, 'var', 'run'))
createPrivateDirectory(os.path.join(self.instance_root, 'etc'))
createPrivateDirectory(self.supervisord_configuration_directory)
# Creates supervisord configuration
updateFile(self.supervisord_configuration_path,
pkg_resources.resource_stream(__name__,
'templates/supervisord.conf.in').read() % dict(
supervisord_configuration_directory=\
self.supervisord_configuration_directory,
supervisord_socket=os.path.abspath(self.supervisord_socket),
supervisord_loglevel='info',
supervisord_logfile=os.path.abspath(os.path.join(
self.instance_root, 'var', 'log', 'supervisord.log')),
supervisord_logfile_maxbytes='50MB',
supervisord_nodaemon='false',
supervisord_pidfile=os.path.abspath(os.path.join(
self.instance_root, 'var', 'run', 'supervisord.pid')),
supervisord_logfile_backups='10',
watchdog_command = self.getWatchdogLine(),
))
except (WrongPermissionError, PathDoesNotExistError) as error:
raise error
'templates/supervisord.conf.in').read() % {
'supervisord_configuration_directory': self.supervisord_configuration_directory,
'supervisord_socket': os.path.abspath(self.supervisord_socket),
'supervisord_loglevel': 'info',
'supervisord_logfile': os.path.abspath(os.path.join(self.instance_root, 'var', 'log', 'supervisord.log')),
'supervisord_logfile_maxbytes': '50MB',
'supervisord_nodaemon': 'false',
'supervisord_pidfile': os.path.abspath(os.path.join(self.instance_root, 'var', 'run', 'supervisord.pid')),
'supervisord_logfile_backups': '10',
'watchdog_command': self.getWatchdogLine(),
})
def getComputerPartitionList(self):
try:
computer_partition_list = self.computer.getComputerPartitionList()
except socket.error as error:
self.logger.fatal(error)
return self.computer.getComputerPartitionList()
except socket.error as exc:
self.logger.fatal(exc)
raise
return computer_partition_list
def processSoftwareReleaseList(self):
"""Will process each Software Release.
"""
self.checkEnvironmentAndCreateStructure()
logger = logging.getLogger('SoftwareReleases')
logger.info("Processing software releases...")
self.logger.info('Processing software releases...')
# Boolean to know if every instance has correctly been deployed
clean_run = True
for software_release in self.computer.getSoftwareReleaseList():
state = software_release.getState()
try:
software_release_uri = software_release.getURI()
url_hash = md5(software_release_uri).hexdigest()
url_hash = md5digest(software_release_uri)
software_path = os.path.join(self.software_root, url_hash)
software = Software(url=software_release_uri,
software_root=self.software_root,
buildout=self.buildout,
logger=self.logger,
signature_private_key_file=self.signature_private_key_file,
signature_certificate_list=self.signature_certificate_list,
download_binary_cache_url=self.download_binary_cache_url,
......@@ -585,43 +410,40 @@ class Slapgrid(object):
if self.develop or (not os.path.exists(completed_tag) and \
len(self.software_release_filter_list) == 0) or \
url_hash in self.software_release_filter_list or \
url_hash in (md5(uri).hexdigest() for uri in self.software_release_filter_list):
url_hash in (md5digest(uri) for uri in self.software_release_filter_list):
try:
software_release.building()
except NotFoundError:
pass
software.install()
file_descriptor = open(completed_tag, 'w')
file_descriptor.write(time.asctime())
file_descriptor.close()
with open(completed_tag, 'w') as fout:
fout.write(time.asctime())
elif state == 'destroyed':
if os.path.exists(software_path):
logger.info('Destroying %r...' % software_release_uri)
self.logger.info('Destroying %r...' % software_release_uri)
software.destroy()
logger.info('Destroyed %r.' % software_release_uri)
self.logger.info('Destroyed %r.' % software_release_uri)
# Send log before exiting
except (SystemExit, KeyboardInterrupt):
exception = traceback.format_exc()
software_release.error(exception)
software_release.error(traceback.format_exc(), logger=self.logger)
raise
# Buildout failed: send log but don't print it to output (already done)
except BuildoutFailedError as exception:
except BuildoutFailedError as exc:
clean_run = False
try:
software_release.error(exception)
software_release.error(exc, logger=self.logger)
except (SystemExit, KeyboardInterrupt):
raise
except Exception:
exception = traceback.format_exc()
logger.error('Problem during reporting error, continuing:\n' +
exception)
self.logger.error('Problem while reporting error, continuing:\n%s' %
traceback.format_exc())
# For everything else: log it, send it, continue.
except Exception:
exception = traceback.format_exc()
logger.error(exception)
software_release.error(exception)
exc = traceback.format_exc()
self.logger.error(exc)
software_release.error(exc, logger=self.logger)
clean_run = False
else:
if state == 'available':
......@@ -634,7 +456,7 @@ class Slapgrid(object):
software_release.destroyed()
except (NotFoundError, ServerError):
print traceback.format_exc()
logger.info("Finished software releases.")
self.logger.info('Finished software releases.')
# Return success value
if not clean_run:
......@@ -644,12 +466,12 @@ class Slapgrid(object):
def _launchSupervisord(self):
launchSupervisord(self.supervisord_socket,
self.supervisord_configuration_path)
self.supervisord_configuration_path,
logger=self.logger)
def _checkPromises(self, computer_partition):
self.logger.info("Checking promises...")
instance_path = os.path.join(self.instance_root,
computer_partition.getId())
instance_path = os.path.join(self.instance_root, computer_partition.getId())
uid, gid = None, None
stat_info = os.stat(instance_path)
......@@ -662,11 +484,8 @@ class Slapgrid(object):
# Get the list of promises
promise_dir = os.path.join(instance_path, 'etc', 'promise')
if os.path.exists(promise_dir) and os.path.isdir(promise_dir):
cwd = instance_path
promises_list = os.listdir(promise_dir)
# Check whether every promise is kept
for promise in promises_list:
for promise in os.listdir(promise_dir):
promise_present = True
command = [os.path.join(promise_dir, promise)]
......@@ -674,13 +493,13 @@ class Slapgrid(object):
promise = os.path.basename(command[0])
self.logger.info("Checking promise %r.", promise)
kw = dict(stdout=subprocess.PIPE, stderr=subprocess.PIPE,
stdin=subprocess.PIPE)
process_handler = subprocess.Popen(command,
preexec_fn=lambda: dropPrivileges(uid, gid),
cwd=cwd,
env=None if sys.platform == 'cygwin' else {}, **kw)
preexec_fn=lambda: dropPrivileges(uid, gid, logger=self.logger),
cwd=instance_path,
env=None if sys.platform == 'cygwin' else {},
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
stdin=subprocess.PIPE)
process_handler.stdin.flush()
process_handler.stdin.close()
process_handler.stdin = None
......@@ -705,8 +524,6 @@ class Slapgrid(object):
"""
Process a Computer Partition, depending on its state
"""
logger = logging.getLogger('ComputerPartitionProcessing')
computer_partition_id = computer_partition.getId()
# Sanity checks before processing
......@@ -720,7 +537,7 @@ class Slapgrid(object):
(computer_partition_id not in self.computer_partition_filter_list):
return
logger.info('Processing Computer Partition %s...' % computer_partition_id)
self.logger.info('Processing Computer Partition %s.' % computer_partition_id)
instance_path = os.path.join(self.instance_root, computer_partition_id)
......@@ -742,13 +559,16 @@ class Slapgrid(object):
# Try to process it anyway, it may need to be deleted.
software_url = None
try:
software_path = os.path.join(self.software_root,
getSoftwareUrlHash(software_url))
software_path = os.path.join(self.software_root, md5digest(software_url))
except TypeError:
# Problem with instance: SR URI not set.
# Try to process it anyway, it may need to be deleted.
software_path = None
self.logger.info(' Software URL: %s' % software_url)
self.logger.info(' Software path: %s' % software_path)
self.logger.info(' Instance path: %s' % instance_path)
periodicity = self.maximum_periodicity
if software_path:
# Get periodicity from periodicity file if not forced
......@@ -759,8 +579,7 @@ class Slapgrid(object):
periodicity = int(open(periodicity_path).read())
except ValueError:
os.remove(periodicity_path)
exception = traceback.format_exc()
logger.error(exception)
self.logger.error(traceback.format_exc())
# Check if timestamp from server is more recent than local one.
# If not: it's not worth processing this partition (nothing has
......@@ -786,8 +605,7 @@ class Slapgrid(object):
os.remove(timestamp_path)
except ValueError:
os.remove(timestamp_path)
exception = traceback.format_exc()
logger.error(exception)
self.logger.error(traceback.format_exc())
local_partition = Partition(
software_path=software_path,
......@@ -802,7 +620,8 @@ class Slapgrid(object):
server_url=self.master_url,
software_release_url=software_url,
certificate_repository_path=self.certificate_repository_path,
buildout=self.buildout)
buildout=self.buildout,
logger=self.logger)
computer_partition_state = computer_partition.getState()
if computer_partition_state == COMPUTER_PARTITION_STARTED_STATE:
......@@ -826,15 +645,14 @@ class Slapgrid(object):
try:
computer_partition.stopped()
except (SystemExit, KeyboardInterrupt):
exception = traceback.format_exc()
computer_partition.error(exception)
computer_partition.error(traceback.format_exc(), logger=self.logger)
raise
except Exception:
pass
else:
error_string = "Computer Partition %r has unsupported state: %s" % \
(computer_partition_id, computer_partition_state)
computer_partition.error(error_string)
computer_partition.error(error_string, logger=self.logger)
raise NotImplementedError(error_string)
# If partition has been successfully processed, write timestamp
......@@ -849,7 +667,6 @@ class Slapgrid(object):
"""
Try to filter valid partitions to be processed from free partitions.
"""
logger = logging.getLogger('ComputerPartitionProcessing')
filtered_computer_partition_list = []
for computer_partition in computer_partition_list:
try:
......@@ -880,32 +697,29 @@ class Slapgrid(object):
# Send log before exiting
except (SystemExit, KeyboardInterrupt):
exception = traceback.format_exc()
computer_partition.error(exception)
computer_partition.error(traceback.format_exc(), logger=self.logger)
raise
# Buildout failed: send log but don't print it to output (already done)
except BuildoutFailedError, exception:
except BuildoutFailedError as exc:
try:
computer_partition.error(exception)
computer_partition.error(exc, logger=self.logger)
except (SystemExit, KeyboardInterrupt):
raise
except Exception:
exception = traceback.format_exc()
logger.error('Problem during reporting error, continuing:\n' +
exception)
self.logger.error('Problem during reporting error, continuing:\n%s' %
traceback.format_exc())
# For everything else: log it, send it, continue.
except Exception as exception:
logger.error(traceback.format_exc())
except Exception as exc:
self.logger.error(traceback.format_exc())
try:
computer_partition.error(exception)
computer_partition.error(exc, logger=self.logger)
except (SystemExit, KeyboardInterrupt):
raise
except Exception:
exception = traceback.format_exc()
logger.error('Problem during reporting error, continuing:\n' +
exception)
self.logger.error('Problem during reporting error, continuing:\n%s' %
traceback.format_exc())
return filtered_computer_partition_list
......@@ -913,8 +727,7 @@ class Slapgrid(object):
"""
Will start supervisord and process each Computer Partition.
"""
logger = logging.getLogger('ComputerPartitionProcessing')
logger.info('Processing computer partitions...')
self.logger.info('Processing computer partitions...')
# Prepares environment
self.checkEnvironmentAndCreateStructure()
self._launchSupervisord()
......@@ -938,48 +751,44 @@ class Slapgrid(object):
# Send log before exiting
except (SystemExit, KeyboardInterrupt):
exception = traceback.format_exc()
computer_partition.error(exception)
computer_partition.error(traceback.format_exc(), logger=self.logger)
raise
except Slapgrid.PromiseError as exception:
except Slapgrid.PromiseError as exc:
clean_run_promise = False
try:
logger.error(exception)
computer_partition.error(exception)
self.logger.error(exc)
computer_partition.error(exc, logger=self.logger)
except (SystemExit, KeyboardInterrupt):
raise
except Exception:
exception = traceback.format_exc()
logger.error('Problem during reporting error, continuing:\n' +
exception)
self.logger.error('Problem during reporting error, continuing:\n%s' %
traceback.format_exc())
# Buildout failed: send log but don't print it to output (already done)
except BuildoutFailedError, exception:
except BuildoutFailedError as exc:
clean_run = False
try:
computer_partition.error(exception)
computer_partition.error(exc, logger=self.logger)
except (SystemExit, KeyboardInterrupt):
raise
except Exception:
exception = traceback.format_exc()
logger.error('Problem during reporting error, continuing:\n' +
exception)
self.logger.error('Problem during reporting error, continuing:\n%s' %
traceback.format_exc())
# For everything else: log it, send it, continue.
except Exception as exception:
except Exception as exc:
clean_run = False
logger.error(traceback.format_exc())
self.logger.error(traceback.format_exc())
try:
computer_partition.error(exception)
computer_partition.error(exc, logger=self.logger)
except (SystemExit, KeyboardInterrupt):
raise
except Exception:
exception = traceback.format_exc()
logger.error('Problem during reporting error, continuing:\n' +
exception)
self.logger.error('Problem during reporting error, continuing:\n%s' %
traceback.format_exc())
logger.info("Finished computer partitions.")
self.logger.info('Finished computer partitions.')
# Return success value
if not clean_run:
......@@ -991,9 +800,6 @@ class Slapgrid(object):
def validateXML(self, to_be_validated, xsd_model):
"""Validates a given xml file"""
logger = logging.getLogger('XMLValidating')
#We retrieve the xsd model
xsd_model = StringIO.StringIO(xsd_model)
xmlschema_doc = etree.parse(xsd_model)
......@@ -1001,10 +807,10 @@ class Slapgrid(object):
try:
document = etree.fromstring(to_be_validated)
except (etree.XMLSyntaxError, etree.DocumentInvalid) as e:
logger.info('Failed to parse this XML report : %s\n%s' % \
(to_be_validated, _formatXMLError(e)))
logger.error(_formatXMLError(e))
except (etree.XMLSyntaxError, etree.DocumentInvalid) as exc:
self.logger.info('Failed to parse this XML report : %s\n%s' % \
(to_be_validated, _formatXMLError(exc)))
self.logger.error(_formatXMLError(exc))
return False
if xmlschema.validate(document):
......@@ -1033,17 +839,16 @@ class Slapgrid(object):
for computer_partition_usage in computer_partition_usage_list:
try:
root = etree.fromstring(computer_partition_usage.usage)
except UnicodeError, e:
self.logger.info("Failed to read %s." % (
computer_partition_usage.usage))
except UnicodeError as exc:
self.logger.info("Failed to read %s." % computer_partition_usage.usage)
self.logger.error(UnicodeError)
raise UnicodeError("Failed to read %s: %s" % (computer_partition_usage.usage, e))
except (etree.XMLSyntaxError, etree.DocumentInvalid) as e:
raise UnicodeError("Failed to read %s: %s" % (computer_partition_usage.usage, exc))
except (etree.XMLSyntaxError, etree.DocumentInvalid) as exc:
self.logger.info("Failed to parse %s." % (computer_partition_usage.usage))
self.logger.error(e)
raise _formatXMLError(e)
except Exception, e:
raise Exception("Failed to generate XML report: %s" % e)
self.logger.error(exc)
raise _formatXMLError(exc)
except Exception as exc:
raise Exception("Failed to generate XML report: %s" % exc)
for movement in root.findall('movement'):
xml.append('<movement>')
......@@ -1067,8 +872,7 @@ class Slapgrid(object):
slap_computer_usage = self.slap.registerComputer(self.computer_id)
computer_partition_usage_list = []
logger = logging.getLogger('UsageReporting')
logger.info("Aggregating and sending usage reports...")
self.logger.info('Aggregating and sending usage reports...')
#We retrieve XSD models
try:
......@@ -1135,26 +939,26 @@ class Slapgrid(object):
#stat sys call to get statistics informations
uid = stat_info.st_uid
gid = stat_info.st_gid
kw = dict(stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
process_handler = SlapPopen(invocation_list,
preexec_fn=lambda: dropPrivileges(uid, gid),
preexec_fn=lambda: dropPrivileges(uid, gid, logger=self.logger),
cwd=os.path.join(instance_path, 'etc', 'report'),
env=None, **kw)
env=None,
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT,
logger=self.logger)
if process_handler.returncode is None:
process_handler.kill()
if process_handler.returncode != 0:
clean_run = False
failed_script_list.append("Script %r failed." % script)
logger.warning("Failed to run %r" % invocation_list)
self.logger.warning('Failed to run %r' % invocation_list)
if len(failed_script_list):
computer_partition.error('\n'.join(failed_script_list))
computer_partition.error('\n'.join(failed_script_list), logger=self.logger)
# Whatever happens, don't stop processing other instances
except Exception:
computer_partition_id = computer_partition.getId()
exception = traceback.format_exc()
issue = "Cannot run usage script(s) for %r: %s" % (
computer_partition_id, exception)
logger.info(issue)
self.logger.info('Cannot run usage script(s) for %r: %s' % (
computer_partition.getId(),
traceback.format_exc()))
#Now we loop through the different computer partitions to report
report_usage_issue_cp_list = []
......@@ -1170,20 +974,17 @@ class Slapgrid(object):
filename_list = os.listdir(dir_reports)
else:
filename_list = []
#logger.debug('name List %s' % filename_list)
usage = ''
#self.logger.debug('name List %s' % filename_list)
for filename in filename_list:
file_path = os.path.join(dir_reports, filename)
if os.path.exists(file_path):
usage_file = open(file_path, 'r')
usage = usage_file.read()
usage_file.close()
usage = open(file_path, 'r').read()
#We check the validity of xml content of each reports
if not self.validateXML(usage, partition_consumption_model):
logger.info('WARNING: The XML file %s generated by slapreport is '
self.logger.info('WARNING: The XML file %s generated by slapreport is '
'not valid - This report is left as is at %s where you can '
'inspect what went wrong ' % (filename, dir_reports))
# Warn the SlapOS Master that a partition generates corrupted xml
......@@ -1195,7 +996,7 @@ class Slapgrid(object):
computer_partition_usage_list.append(computer_partition_usage)
filename_delete_list.append(filename)
else:
logger.debug("Usage report %r not found, ignored" % file_path)
self.logger.debug('Usage report %r not found, ignored' % file_path)
#After sending the aggregated file we remove all the valid xml reports
for filename in filename_delete_list:
......@@ -1203,15 +1004,14 @@ class Slapgrid(object):
# Whatever happens, don't stop processing other instances
except Exception:
computer_partition_id = computer_partition.getId()
exception = traceback.format_exc()
issue = "Cannot run usage script(s) for %r: %s" % (
computer_partition_id, exception)
logger.info(issue)
self.logger.info('Cannot run usage script(s) for %r: %s' % (
computer_partition.getId(),
traceback.format_exc()))
for computer_partition_usage in computer_partition_usage_list:
logger.info('computer_partition_usage_list : %s - %s' % \
(computer_partition_usage.usage, computer_partition_usage.getId()))
self.logger.info('computer_partition_usage_list: %s - %s' % (
computer_partition_usage.usage,
computer_partition_usage.getId()))
#If there is, at least, one report
if computer_partition_usage_list != []:
......@@ -1219,22 +1019,21 @@ class Slapgrid(object):
#We generate the final XML report with asXML method
computer_consumption = self.asXML(computer_partition_usage_list)
logger.info('Final xml report : %s' % computer_consumption)
self.logger.info('Final xml report: %s' % computer_consumption)
#We test the XML report before sending it
if self.validateXML(computer_consumption, computer_consumption_model):
logger.info('XML file generated by asXML is valid')
self.logger.info('XML file generated by asXML is valid')
slap_computer_usage.reportUsage(computer_consumption)
else:
logger.info('XML file generated by asXML is not valid !')
self.logger.info('XML file generated by asXML is not valid !')
raise ValueError('XML file generated by asXML is not valid !')
except Exception:
computer_partition_id = computer_partition.getId()
exception = traceback.format_exc()
issue = "Cannot report usage for %r: %s" % (computer_partition_id,
exception)
logger.info(issue)
computer_partition.error(issue)
issue = "Cannot report usage for %r: %s" % (
computer_partition.getId(),
traceback.format_exc())
self.logger.info(issue)
computer_partition.error(issue, logger=self.logger)
report_usage_issue_cp_list.append(computer_partition_id)
for computer_partition in computer_partition_list:
......@@ -1243,11 +1042,11 @@ class Slapgrid(object):
computer_partition_id = computer_partition.getId()
try:
software_url = computer_partition.getSoftwareRelease().getURI()
software_path = os.path.join(self.software_root,
getSoftwareUrlHash(software_url))
software_path = os.path.join(self.software_root, md5digest(software_url))
except (NotFoundError, TypeError):
software_url = None
software_path = None
local_partition = Partition(
software_path=software_path,
instance_path=os.path.join(self.instance_root,
......@@ -1263,42 +1062,40 @@ class Slapgrid(object):
software_release_url=software_url,
certificate_repository_path=self.certificate_repository_path,
buildout=self.buildout,
)
logger=self.logger)
local_partition.stop()
try:
computer_partition.stopped()
except (SystemExit, KeyboardInterrupt):
exception = traceback.format_exc()
computer_partition.error(exception)
computer_partition.error(traceback.format_exc(), logger=self.logger)
raise
except Exception:
pass
if computer_partition.getId() in report_usage_issue_cp_list:
logger.info('Ignoring destruction of %r, as not report usage was '
'sent' % computer_partition.getId())
self.logger.info('Ignoring destruction of %r, as no report usage was sent' %
computer_partition.getId())
continue
local_partition.destroy()
except (SystemExit, KeyboardInterrupt):
exception = traceback.format_exc()
computer_partition.error(exception)
computer_partition.error(traceback.format_exc(), logger=self.logger)
raise
except Exception:
clean_run = False
exception = traceback.format_exc()
computer_partition.error(exception)
logger.error(exception)
exc = traceback.format_exc()
computer_partition.error(exc, logger=self.logger)
self.logger.error(exc)
try:
computer_partition.destroyed()
except slap.NotFoundError:
logger.debug('Ignored slap error while trying to inform about '
except NotFoundError:
self.logger.debug('Ignored slap error while trying to inform about '
'destroying not fully configured Computer Partition %r' %
computer_partition.getId())
except ServerError as server_error:
logger.debug('Ignored server error while trying to inform about '
'destroying Computer Partition %r. Error is :\n%r' %
self.logger.debug('Ignored server error while trying to inform about '
'destroying Computer Partition %r. Error is:\n%r' %
(computer_partition.getId(), server_error.args[0]))
logger.info("Finished usage reports.")
self.logger.info('Finished usage reports.')
# Return success value
if not clean_run:
......
slapos/grid/svcbackend.py
View file @ 5f0924ff
......@@ -28,18 +28,16 @@
#
##############################################################################
from supervisor import xmlrpc
import time
from utils import SlapPopen
import logging
import os
import sys
import xmlrpclib
from optparse import OptionParser
import ConfigParser
import socket as socketlib
import subprocess
from supervisor import xmlrpc
from slapos.grid.utils import SlapPopen
def getSupervisorRPC(socket):
supervisor_transport = xmlrpc.SupervisorTransport('', '',
......@@ -49,20 +47,19 @@ def getSupervisorRPC(socket):
return getattr(server_proxy, 'supervisor')
def launchSupervisord(socket, configuration_file):
logger = logging.getLogger('SVCBackend')
supervisor = getSupervisorRPC(socket)
def launchSupervisord(socket, configuration_file, logger):
if os.path.exists(socket):
trynum = 1
while trynum < 6:
try:
supervisor = getSupervisorRPC(socket)
status = supervisor.getState()
except xmlrpclib.Fault as e:
if e.faultCode == 6 and e.faultString == 'SHUTDOWN_STATE':
logger.info('Supervisor in shutdown procedure, will check again later.')
trynum += 1
time.sleep(2 * trynum)
except Exception:
except Exception as e:
# In case if there is problem with connection, assume that supervisord
# is not running and try to run it
break
......@@ -90,10 +87,14 @@ def launchSupervisord(socket, configuration_file):
env={},
executable=sys.executable,
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT)
stderr=subprocess.STDOUT,
logger=logger)
result = supervisord_popen.communicate()[0]
if supervisord_popen.returncode == 0:
logger.info('Supervisord command invoked with: %s' % result)
if supervisord_popen.returncode:
logger.warning('Supervisord unknown problem: %s' % result)
return
try:
default_timeout = socketlib.getdefaulttimeout()
current_timeout = 1
......@@ -101,13 +102,14 @@ def launchSupervisord(socket, configuration_file):
while trynum < 6:
try:
socketlib.setdefaulttimeout(current_timeout)
supervisor = getSupervisorRPC(socket)
status = supervisor.getState()
if status['statename'] == 'RUNNING' and status['statecode'] == 1:
return
logger.warning('Wrong status name %(statename)r and code '
'%(statecode)r, trying again' % status)
trynum += 1
except Exception:
except Exception as e:
current_timeout = 5 * trynum
trynum += 1
else:
......@@ -116,59 +118,3 @@ def launchSupervisord(socket, configuration_file):
logger.warning('Issue while checking supervisord.')
finally:
socketlib.setdefaulttimeout(default_timeout)
else:
log_message = 'Supervisord unknown problem: %s' % result
logger.warning(log_message)
def getOptionDict(*argument_tuple):
usage = """
Typical usage:
* %prog CONFIGURATION_FILE [arguments passed to supervisor]
""".strip()
parser = OptionParser(usage=usage)
# Parses arguments
if argument_tuple:
(argument_option_instance, argument_list) = parser.parse_args(list(argument_tuple))
else:
# No arguments given to entry point : we parse sys.argv.
(argument_option_instance, argument_list) = parser.parse_args()
if not argument_list:
parser.error("Configuration file is obligatory. Consult documentation by "
"calling with -h.")
configuration_file = argument_list[0]
if not os.path.exists(configuration_file):
parser.error("Could not read configuration file : %s" % configuration_file)
slapgrid_configuration = ConfigParser.SafeConfigParser()
slapgrid_configuration.read(configuration_file)
# Merges the two dictionnaries
option_dict = dict(slapgrid_configuration.items("slapos"))
# Supervisord configuration location
option_dict.setdefault('supervisord_configuration_path',
os.path.join(option_dict['instance_root'], 'etc', 'supervisord.conf'))
# Supervisord socket
option_dict.setdefault('supervisord_socket',
os.path.join(option_dict['instance_root'], 'supervisord.socket'))
return option_dict, argument_list[1:]
def supervisorctl(*argument_tuple):
option_dict, args = getOptionDict(*argument_tuple)
import supervisor.supervisorctl
launchSupervisord(option_dict['supervisord_socket'],
option_dict['supervisord_configuration_path'])
supervisor.supervisorctl.main(args=['-c',
option_dict['supervisord_configuration_path']] + args)
def supervisord(*argument_tuple):
option_dict, _ = getOptionDict(*argument_tuple)
launchSupervisord(option_dict['supervisord_socket'],
option_dict['supervisord_configuration_path'])
slapos/grid/utils.py
View file @ 5f0924ff
# -*- coding: utf-8 -*-
# vim: set et sts=2:
##############################################################################
#
# Copyright (c) 2010, 2011, 2012 Vifib SARL and Contributors.
......@@ -27,21 +28,20 @@
#
##############################################################################
import logging
import grp
import hashlib
import os
import pkg_resources
import pwd
import stat
import subprocess
import sys
import pwd
import grp
from exception import BuildoutFailedError, WrongPermissionError
from hashlib import md5
from slapos.grid.exception import BuildoutFailedError, WrongPermissionError
# Such umask by default will create paths with full permission
# for user, non writable by group and not accessible by others
SAFE_UMASK = 027
SAFE_UMASK = 0o27
PYTHON_ENVIRONMENT_REMOVE_LIST = [
'PYTHONHOME',
......@@ -94,6 +94,7 @@ class SlapPopen(subprocess.Popen):
log.
"""
def __init__(self, *args, **kwargs):
logger = kwargs.pop('logger')
kwargs.update(stdin=subprocess.PIPE)
if sys.platform == 'cygwin' and kwargs.get('env') == {}:
kwargs['env'] = None
......@@ -102,25 +103,22 @@ class SlapPopen(subprocess.Popen):
self.stdin.close()
self.stdin = None
logger = logging.getLogger('SlapProcessManager')
# XXX-Cedric: this algorithm looks overkill for simple logging.
output_lines = []
while True:
line = self.stdout.readline()
if line == '' and self.poll() != None:
if line == '' and self.poll() is not None:
break
output_lines.append(line)
if line[-1:] == '\n':
line = line[:-1]
logger.info(line)
logger.info(line.rstrip('\n'))
self.output = ''.join(output_lines)
def getSoftwareUrlHash(url):
return md5(url).hexdigest()
def md5digest(url):
return hashlib.md5(url).hexdigest()
def getCleanEnvironment(home_path='/tmp'):
logger = logging.getLogger('CleanEnvironment')
def getCleanEnvironment(logger, home_path='/tmp'):
changed_env = {}
removed_env = []
env = os.environ.copy()
......@@ -137,46 +135,41 @@ def getCleanEnvironment(home_path='/tmp'):
return env
def setRunning(pid_file):
def setRunning(logger, pidfile):
"""Creates a pidfile. If a pidfile already exists, we exit"""
logger = logging.getLogger('Slapgrid')
if os.path.exists(pid_file):
# Pid file is present
# XXX might use http://code.activestate.com/recipes/577911-context-manager-for-a-daemon-pid-file/
if os.path.exists(pidfile):
try:
pid = int(open(pid_file, 'r').readline())
pid = int(open(pidfile, 'r').readline())
except ValueError:
pid = None
# XXX This could use psutil library.
if pid is not None and os.path.exists("/proc/%s" % pid):
# In case process is present, ignore.
if pid and os.path.exists("/proc/%s" % pid):
logger.info('New slapos process started, but another slapos '
'process is aleady running with pid %s, exiting.' % pid)
sys.exit(10)
logger.info('Existing pid file %r was stale one, overwritten' % pid_file)
logger.info('Existing pid file %r was stale, overwritten' % pidfile)
# Start new process
write_pid(pid_file)
write_pid(logger, pidfile)
def setFinished(pid_file):
def setFinished(pidfile):
try:
os.remove(pid_file)
os.remove(pidfile)
except OSError:
pass
def write_pid(pid_file):
logger = logging.getLogger('Slapgrid')
pid = os.getpid()
def write_pid(logger, pidfile):
try:
f = open(pid_file, 'w')
f.write('%s' % pid)
f.close()
with open(pidfile, 'w') as fout:
fout.write('%s' % os.getpid())
except (IOError, OSError):
logger.critical('slapgrid could not write pidfile %s' % pid_file)
logger.critical('slapgrid could not write pidfile %s' % pidfile)
raise
def dropPrivileges(uid, gid):
def dropPrivileges(uid, gid, logger):
"""Drop privileges to uid, gid if current uid is 0
Do tests to check if dropping was successful and that no system call is able
......@@ -184,14 +177,13 @@ def dropPrivileges(uid, gid):
Does nothing in case if uid and gid are not 0
"""
logger = logging.getLogger('dropPrivileges')
# XXX-Cedric: remove format / just do a print, otherwise formatting is done
# twice
current_uid, current_gid = os.getuid(), os.getgid()
if uid == 0 or gid == 0:
raise OSError('Dropping privileges to uid = %r or ' \
'gid = %r is too dangerous' % (uid, gid))
if not(current_uid == 0 and current_gid == 0):
if current_uid or current_gid:
logger.debug('Running as uid = %r, gid = %r, dropping not needed and not '
'possible' % (current_uid, current_gid))
return
......@@ -209,7 +201,7 @@ def dropPrivileges(uid, gid):
uid, gid, group_list)
new_uid, new_gid, new_group_list = os.getuid(), os.getgid(), os.getgroups()
if not (new_uid == uid and new_gid == gid and set(new_group_list) == group_list):
raise OSError('%s new_uid = %r and new_gid = %r and ' \
raise OSError('%s new_uid = %r and new_gid = %r and '
'new_group_list = %r which is fatal.'
% (message_pre,
new_uid,
......@@ -236,18 +228,16 @@ def dropPrivileges(uid, gid):
logger.debug('Succesfully dropped privileges to uid=%r gid=%r' % (uid, gid))
def bootstrapBuildout(path, buildout=None,
def bootstrapBuildout(path, logger, buildout=None,
additional_buildout_parametr_list=None):
if additional_buildout_parametr_list is None:
additional_buildout_parametr_list = []
logger = logging.getLogger('BuildoutManager')
# Reads uid/gid of path, launches buildout with thoses privileges
stat_info = os.stat(path)
uid = stat_info.st_uid
gid = stat_info.st_gid
invocation_list = [sys.executable, '-S']
kw = dict()
if buildout is not None:
invocation_list.append(buildout)
invocation_list.extend(additional_buildout_parametr_list)
......@@ -274,25 +264,26 @@ def bootstrapBuildout(path, buildout=None,
logger.debug('Set umask from %03o to %03o' % (umask, SAFE_UMASK))
logger.debug('Invoking: %r in directory %r' % (' '.join(invocation_list),
path))
kw.update(stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
process_handler = SlapPopen(invocation_list,
preexec_fn=lambda: dropPrivileges(uid, gid),
cwd=path, **kw)
preexec_fn=lambda: dropPrivileges(uid, gid, logger=logger),
cwd=path,
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT,
logger=logger)
if process_handler.returncode is None or process_handler.returncode != 0:
message = 'Failed to run buildout profile in directory %r' % (path)
logger.error(message)
raise BuildoutFailedError('%s:\n%s\n' % (message, process_handler.output))
except OSError as error:
raise BuildoutFailedError(error)
except OSError as exc:
raise BuildoutFailedError(exc)
finally:
old_umask = os.umask(umask)
logger.debug('Restore umask from %03o to %03o' % (old_umask, umask))
def launchBuildout(path, buildout_binary,
def launchBuildout(path, buildout_binary, logger,
additional_buildout_parametr_list=None):
""" Launches buildout."""
logger = logging.getLogger('BuildoutManager')
if additional_buildout_parametr_list is None:
additional_buildout_parametr_list = []
# Reads uid/gid of path, launches buildout with thoses privileges
......@@ -300,9 +291,7 @@ def launchBuildout(path, buildout_binary,
uid = stat_info.st_uid
gid = stat_info.st_gid
# Extract python binary to prevent shebang size limit
file = open(buildout_binary, 'r')
line = file.readline()
file.close()
line = open(buildout_binary, 'r').readline()
invocation_list = []
if line.startswith('#!'):
line = line[2:]
......@@ -316,42 +305,44 @@ def launchBuildout(path, buildout_binary,
logger.debug('Set umask from %03o to %03o' % (umask, SAFE_UMASK))
logger.debug('Invoking: %r in directory %r' % (' '.join(invocation_list),
path))
kw = dict(stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
process_handler = SlapPopen(invocation_list,
preexec_fn=lambda: dropPrivileges(uid, gid), cwd=path,
env=getCleanEnvironment(pwd.getpwuid(uid).pw_dir), **kw)
preexec_fn=lambda: dropPrivileges(uid, gid, logger=logger),
cwd=path,
env=getCleanEnvironment(logger=logger,
home_path=path),
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT,
logger=logger)
if process_handler.returncode is None or process_handler.returncode != 0:
message = 'Failed to run buildout profile in directory %r' % (path)
logger.error(message)
raise BuildoutFailedError('%s:\n%s\n' % (message, process_handler.output))
except OSError as error:
raise BuildoutFailedError(error)
except OSError as exc:
raise BuildoutFailedError(exc)
finally:
old_umask = os.umask(umask)
logger.debug('Restore umask from %03o to %03o' % (old_umask, umask))
def updateFile(file_path, content, mode='0600'):
def updateFile(file_path, content, mode=0o600):
"""Creates an executable with "content" as content."""
altered = False
if not (os.path.isfile(file_path)) or \
not(hashlib.md5(open(file_path).read()).digest() ==\
hashlib.md5(content).digest()):
with open(file_path, 'w') as fout:
fout.write(content)
altered = True
file_file = open(file_path, 'w')
file_file.write(content)
file_file.flush()
file_file.close()
os.chmod(file_path, stat.S_IREAD | stat.S_IWRITE | stat.S_IEXEC)
if oct(stat.S_IMODE(os.stat(file_path).st_mode)) != mode:
os.chmod(file_path, int(mode, 8))
if stat.S_IMODE(os.stat(file_path).st_mode) != mode:
os.chmod(file_path, mode)
altered = True
return altered
def updateExecutable(executable_path, content):
"""Creates an executable with "content" as content."""
return updateFile(executable_path, content, '0700')
return updateFile(executable_path, content, 0o700)
def createPrivateDirectory(path):
......@@ -359,8 +350,8 @@ def createPrivateDirectory(path):
if not os.path.isdir(path):
os.mkdir(path)
os.chmod(path, stat.S_IREAD | stat.S_IWRITE | stat.S_IEXEC)
permission = oct(stat.S_IMODE(os.stat(path).st_mode))
if permission not in ('0700'):
raise WrongPermissionError('Wrong permissions in %s ' \
': is %s, should be 0700'
permission = stat.S_IMODE(os.stat(path).st_mode)
if permission != 0o700:
raise WrongPermissionError('Wrong permissions in %s: ' \
'is 0%o, should be 0700'
% (path, permission))
slapos/proxy/__init__.py
View file @ 5f0924ff
# -*- coding: utf-8 -*-
# vim: set et sts=2:
##############################################################################
#
# Copyright (c) 2010, 2011, 2012 Vifib SARL and Contributors.
......@@ -27,113 +28,38 @@
#
##############################################################################
import os
import sys
from optparse import OptionParser, Option
import logging
import logging.handlers
import ConfigParser
class Parser(OptionParser):
"""
Parse all arguments.
"""
def __init__(self, usage=None, version=None):
"""
Initialize all options possibles.
"""
OptionParser.__init__(self, usage=usage, version=version,
option_list=[
Option("-l", "--log_file",
help="The path to the log file used by the script.",
type=str),
Option("-v", "--verbose",
default=False,
action="store_true",
help="Verbose output."),
Option("-c", "--console",
default=False,
action="store_true",
help="Console output."),
Option("-u", "--database-uri",
type=str,
help="URI for sqlite database"),
])
class ProxyConfig(object):
def __init__(self, logger):
self.logger = logger
def check_args(self):
"""
Check arguments
"""
(options, args) = self.parse_args()
if len(args) != 1:
self.error("Incorrect number of arguments")
return options, args[0]
class Config:
def setConfig(self, option_dict, configuration_file_path):
"""
Set options given by parameters.
"""
def mergeConfig(self, args, configp):
# Set options parameters
for option, value in option_dict.__dict__.items():
for option, value in args.__dict__.items():
setattr(self, option, value)
# Load configuration file
configuration_parser = ConfigParser.SafeConfigParser()
configuration_parser.read(configuration_file_path)
# Merges the arguments and configuration
for section in ("slapproxy", "slapos"):
configuration_dict = dict(configuration_parser.items(section))
configuration_dict = dict(configp.items(section))
for key in configuration_dict:
if not getattr(self, key, None):
setattr(self, key, configuration_dict[key])
# set up logging
self.logger = logging.getLogger("slapproxy")
self.logger.setLevel(logging.INFO)
if self.console:
self.logger.addHandler(logging.StreamHandler())
def setConfig(self):
if not self.database_uri:
raise ValueError('database-uri is required.')
if self.log_file:
if not os.path.isdir(os.path.dirname(self.log_file)):
# fallback to console only if directory for logs does not exists and
# continue to run
raise ValueError('Please create directory %r to store %r log file' % (
os.path.dirname(self.log_file), self.log_file))
else:
file_handler = logging.FileHandler(self.log_file)
file_handler.setFormatter(logging.Formatter("%(asctime)s - %(name)s - %(levelname)s - %(message)s"))
self.logger.addHandler(file_handler)
self.logger.info('Configured logging to file %r' % self.log_file)
self.logger.info("Started.")
if self.verbose:
self.logger.setLevel(logging.DEBUG)
self.logger.debug("Verbose mode enabled.")
def run(config):
from views import app
app.config['computer_id'] = config.computer_id
app.config['DATABASE_URI'] = config.database_uri
app.run(host=config.host, port=int(config.port))
def main():
"Run default configuration."
usage = "usage: %s [options] CONFIGURATION_FILE" % sys.argv[0]
try:
# Parse arguments
config = Config()
config.setConfig(*Parser(usage=usage).check_args())
def do_proxy(conf):
from slapos.proxy.views import app
for handler in conf.logger.handlers:
app.logger.addHandler(handler)
app.logger.setLevel(logging.INFO)
app.config['computer_id'] = conf.computer_id
app.config['DATABASE_URI'] = conf.database_uri
app.run(host=conf.host, port=int(conf.port))
run(config)
return_code = 0
except SystemExit, err:
# Catch exception raise by optparse
return_code = err
sys.exit(return_code)
slapos/register/register.py
View file @ 5f0924ff
# -*- coding: utf-8 -*-
# vim: set et sts=2:
##############################################################################
#
# Copyright (c) 2012 Vifib SARL and Contributors. All Rights Reserved.
......@@ -26,12 +27,11 @@
#
##############################################################################
# XXX dry_run will happily register a new node on the slapos master. Isn't it supposed to be no-op?
import base64
import ConfigParser
from getpass import getpass
import logging
from optparse import OptionParser, Option
import getpass
import os
import shutil
import stat
......@@ -40,119 +40,40 @@ import tempfile
import urllib2
class SlapError(Exception):
"""
Slap error
"""
def __init__(self, message):
self.msg = message
class UsageError(SlapError):
pass
class ExecError(SlapError):
pass
def authenticate(request, login, password):
auth = '%s:%s' % (login, password)
authheader = 'Basic %s' % auth.encode('base64').rstrip()
request.add_header('Authorization', authheader)
class Parser(OptionParser):
"""
Parse all arguments.
"""
def __init__(self, usage=None, version=None):
"""
Initialize all options possibles.
"""
OptionParser.__init__(self, usage=usage, version=version,
option_list=[
Option("--interface-name",
help="Interface name to access internet",
default='eth0',
type=str),
Option("--master-url",
help="URL of SlapOS master",
default='https://slap.vifib.com',
type=str),
Option("--master-url-web",
help="URL of SlapOS Master webservice to register certificates",
default='https://www.slapos.org',
type=str),
Option("--partition-number",
help="Number of partition on computer",
default='10',
type=int),
Option("--ipv4-local-network",
help="Base of ipv4 local network",
default='10.0.0.0/16',
type=str),
Option("--ipv6-interface",
help="Interface name to get ipv6",
default='',
type=str),
Option("--login",
help="User login on SlapOS Master webservice",
default=None,
type=str),
Option("--password",
help="User password on SlapOs Master webservice",
default=None,
type=str),
Option("-t", "--create-tap",
help="""Will trigger creation of one virtual "tap" interface per \
Partition and attach it to primary interface. Requires primary interface to be \
a bridge. defaults to false. Needed to host virtual machines.""",
default=False,
action="store_true"),
Option("-n", "--dry-run",
help="Simulate the execution steps",
default=False,
action="store_true"),
])
def check_args(self):
"""
Check arguments
"""
(options, args) = self.parse_args()
if len(args) != 1:
self.error("Incorrect number of arguments")
node_name = args[0]
if options.password != None and options.login == None :
self.error("Please enter your login with your password")
return options, node_name
def get_login():
"""Get user id and encode it for basic identification"""
login = raw_input("SlapOS Master Login: ")
password = getpass()
identification = base64.encodestring('%s:%s' % (login, password))[:-1]
return identification
def check_login(identification, master_url_web):
def check_credentials(url, login, password):
"""Check if logged correctly on SlapOS Master"""
request = urllib2.Request(master_url_web)
# Prepare header for basic authentification
authheader = "Basic %s" % identification
request.add_header("Authorization", authheader)
home_page_url = urllib2.urlopen(request).read()
if 'Logout' in home_page_url:
return 1
else : return 0
request = urllib2.Request(url)
authenticate(request, login, password)
return 'Logout' in urllib2.urlopen(request).read()
def get_certificates(identification, node_name, master_url_web):
def get_certificates(master_url_web, node_name, login, password, logger):
"""Download certificates from SlapOS Master"""
register_server_url = '/'.join([master_url_web, ("add-a-server/WebSection_registerNewComputer?dialog_id=WebSection_viewServerInformationDialog&dialog_method=WebSection_registerNewComputer&title={}&object_path=/erp5/web_site_module/hosting/add-a-server&update_method=&cancel_url=https%3A//www.vifib.net/add-a-server/WebSection_viewServerInformationDialog&Base_callDialogMethod=&field_your_title=Essai1&dialog_category=None&form_id=view".format(node_name))])
request = urllib2.Request(register_server_url)
# Prepare header for basic authentification
authheader = "Basic %s" % identification
request.add_header("Authorization", authheader)
url = urllib2.urlopen(request)
page = url.read()
return page
authenticate(request, login, password)
try:
req = urllib2.urlopen(request)
except urllib2.HTTPError as exc:
# raise a readable exception if the computer name is already used,
# instead of an opaque 500 Internal Error.
# this will not work with the new API.
if exc.getcode() == 500:
error_body = exc.read()
if 'Certificate still active.' in error_body:
logger.error('The node name "%s" is already in use. Please change the name, or revoke the active certificate if you want to replace the node.' % node_name)
sys.exit(1)
raise
return req.read()
def parse_certificates(source):
......@@ -161,7 +82,7 @@ def parse_certificates(source):
c_end = source.find("</textarea>", c_start)
k_start = source.find("-----BEGIN PRIVATE KEY-----")
k_end = source.find("</textarea>", k_start)
return [source[c_start:c_end], source[k_start:k_end]]
return source[c_start:c_end], source[k_start:k_end]
def get_computer_name(certificate):
......@@ -170,50 +91,50 @@ def get_computer_name(certificate):
i = certificate.find("/email", k)
return certificate[k:i]
def save_former_config(config):
def save_former_config(conf):
"""Save former configuration if found"""
# Check for config file in /etc/opt/slapos/
if os.path.exists('/etc/opt/slapos/slapos.cfg'):
former_slapos_configuration = '/etc/opt/slapos'
else : former_slapos_configuration = 0
if former_slapos_configuration:
saved_slapos_configuration = former_slapos_configuration + '.old'
former = '/etc/opt/slapos'
else:
return
saved = former + '.old'
while True:
if os.path.exists(saved_slapos_configuration):
print "Slapos configuration detected in %s" % saved_slapos_configuration
if saved_slapos_configuration[len(saved_slapos_configuration) - 1] != 'd' :
saved_slapos_configuration = saved_slapos_configuration[:len(saved_slapos_configuration) - 1] \
+ str(int(saved_slapos_configuration[len(saved_slapos_configuration) - 1]) + 1 )
else :
saved_slapos_configuration += ".1"
else: break
config.logger.info("Former slapos configuration detected in %s moving to %s" % (former_slapos_configuration, saved_slapos_configuration))
shutil.move(former_slapos_configuration, saved_slapos_configuration)
if os.path.exists(saved):
print "Slapos configuration detected in %s" % saved
if saved[-1] != 'd':
saved = saved[:-1] + str(int(saved[-1]) + 1)
else:
saved += '.1'
else:
break
conf.logger.info("Former slapos configuration detected in %s moving to %s" % (former, saved))
shutil.move(former, saved)
def get_slapos_conf_example():
"""
Get slapos.cfg.example and return its path
"""
register_server_url = "http://git.erp5.org/gitweb/slapos.core.git/blob_plain/HEAD:/slapos.cfg.example"
request = urllib2.Request(register_server_url)
url = urllib2.urlopen(request)
page = url.read()
request = urllib2.Request('http://git.erp5.org/gitweb/slapos.core.git/blob_plain/HEAD:/slapos.cfg.example')
req = urllib2.urlopen(request)
_, path = tempfile.mkstemp()
slapos_cfg_example = open(path,'w')
slapos_cfg_example.write(page)
slapos_cfg_example.close()
with open(path, 'w') as fout:
fout.write(req.read())
return path
def slapconfig(config):
def slapconfig(conf):
"""Base Function to configure slapos in /etc/opt/slapos"""
dry_run = config.dry_run
dry_run = conf.dry_run
# Create slapos configuration directory if needed
slap_configuration_directory = os.path.normpath(config.slapos_configuration)
slap_conf_dir = os.path.normpath(conf.slapos_configuration)
# Make sure everybody can read slapos configuration directory:
# Add +x to directories in path
directory = os.path.dirname(slap_configuration_directory)
directory = os.path.dirname(slap_conf_dir)
while True:
if os.path.dirname(directory) == directory:
break
......@@ -221,100 +142,84 @@ def slapconfig(config):
os.chmod(directory, os.stat(directory).st_mode | stat.S_IXGRP | stat.S_IRGRP | stat.S_IXOTH | stat.S_IROTH)
directory = os.path.dirname(directory)
if not os.path.exists(slap_configuration_directory):
config.logger.info ("Creating directory: %s" % slap_configuration_directory)
if not os.path.exists(slap_conf_dir):
conf.logger.info("Creating directory: %s" % slap_conf_dir)
if not dry_run:
os.mkdir(slap_configuration_directory, 0711)
os.mkdir(slap_conf_dir, 0o711)
user_certificate_repository_path = os.path.join(slap_configuration_directory,'ssl')
user_certificate_repository_path = os.path.join(slap_conf_dir, 'ssl')
if not os.path.exists(user_certificate_repository_path):
config.logger.info ("Creating directory: %s" % user_certificate_repository_path)
conf.logger.info("Creating directory: %s" % user_certificate_repository_path)
if not dry_run:
os.mkdir(user_certificate_repository_path, 0711)
os.mkdir(user_certificate_repository_path, 0o711)
key_file = os.path.join(user_certificate_repository_path, 'key')
cert_file = os.path.join(user_certificate_repository_path, 'certificate')
for (src, dst) in [(config.key, key_file), (config.certificate,
cert_file)]:
config.logger.info ("Copying to %r, and setting minimum privileges" % dst)
for src, dst in [
(conf.key, key_file),
(conf.certificate, cert_file)
]:
conf.logger.info("Copying to %r, and setting minimum privileges" % dst)
if not dry_run:
destination = open(dst,'w')
with open(dst, 'w') as destination:
destination.write(''.join(src))
destination.close()
os.chmod(dst, 0600)
os.chmod(dst, 0o600)
os.chown(dst, 0, 0)
certificate_repository_path = os.path.join(slap_configuration_directory, 'ssl', 'partition_pki')
certificate_repository_path = os.path.join(slap_conf_dir, 'ssl', 'partition_pki')
if not os.path.exists(certificate_repository_path):
config.logger.info ("Creating directory: %s" % certificate_repository_path)
conf.logger.info("Creating directory: %s" % certificate_repository_path)
if not dry_run:
os.mkdir(certificate_repository_path, 0711)
os.mkdir(certificate_repository_path, 0o711)
# Put slapos configuration file
slap_configuration_file_location = os.path.join(slap_configuration_directory,
'slapos.cfg')
config.logger.info ("Creating slap configuration: %s"
% slap_configuration_file_location)
slap_conf_file = os.path.join(slap_conf_dir, 'slapos.cfg')
conf.logger.info("Creating slap configuration: %s" % slap_conf_file)
# Get example configuration file
slapos_cfg_example = get_slapos_conf_example()
configuration_example_parser = ConfigParser.RawConfigParser()
configuration_example_parser.read(slapos_cfg_example)
new_configp = ConfigParser.RawConfigParser()
new_configp.read(slapos_cfg_example)
os.remove(slapos_cfg_example)
# prepare slapos section
slaposconfig = dict(
computer_id=config.computer_id, master_url=config.master_url,
key_file=key_file, cert_file=cert_file,
certificate_repository_path=certificate_repository_path)
for key in slaposconfig:
configuration_example_parser.set('slapos', key, slaposconfig[key])
# prepare slapformat
slapformatconfig = dict(
interface_name=config.interface_name,
ipv4_local_network=config.ipv4_local_network,
partition_amount=config.partition_number,
create_tap=config.create_tap
)
for key in slapformatconfig :
configuration_example_parser.set('slapformat', key, slapformatconfig[key])
if not config.ipv6_interface == '':
configuration_example_parser.set('slapformat',
'ipv6_interface',
config.ipv6_interface)
for section, key, value in [
('slapos', 'computer_id', conf.computer_id),
('slapos', 'master_url', conf.master_url),
('slapos', 'key_file', key_file),
('slapos', 'cert_file', cert_file),
('slapos', 'certificate_repository_path', certificate_repository_path),
('slapformat', 'interface_name', conf.interface_name),
('slapformat', 'ipv4_local_network', conf.ipv4_local_network),
('slapformat', 'partition_amount', conf.partition_number),
('slapformat', 'create_tap', conf.create_tap)
]:
new_configp.set(section, key, value)
if conf.ipv6_interface:
new_configp.set('slapformat', 'ipv6_interface', conf.ipv6_interface)
if not dry_run:
slap_configuration_file = open(slap_configuration_file_location, "w")
configuration_example_parser.write(slap_configuration_file)
with open(slap_conf_file, 'w') as fout:
new_configp.write(fout)
conf.logger.info("SlapOS configuration: DONE")
config.logger.info ("SlapOS configuration: DONE")
class RegisterConfig(object):
"""
Class containing all parameters needed for configuration
"""
def __init__(self, logger):
self.logger = logger
# Class containing all parameters needed for configuration
class Config:
def setConfig(self, option_dict, node_name):
def setConfig(self, options):
"""
Set options given by parameters.
"""
# Set options parameters
for option, value in option_dict.__dict__.items():
for option, value in options.__dict__.items():
setattr(self, option, value)
self.node_name = node_name
# Define logger for register
self.logger = logging.getLogger('Register')
self.logger.setLevel(logging.DEBUG)
# create console handler and set level to debug
self.ch = logging.StreamHandler()
self.ch.setLevel(logging.INFO)
# create formatter
self.formatter = logging.Formatter('%(levelname)s - %(message)s')
# add formatter to ch
self.ch.setFormatter(self.formatter)
# add ch to logger
self.logger.addHandler(self.ch)
def COMPConfig(self, slapos_configuration, computer_id, certificate, key):
self.slapos_configuration = slapos_configuration
......@@ -328,66 +233,46 @@ class Config:
self.logger.debug("Number of partition: %s" % self.partition_number)
self.logger.info("Using Interface %s" % self.interface_name)
self.logger.debug("Ipv4 sub network: %s" % self.ipv4_local_network)
self.logger.debug("Ipv6 Interface: %s" %self.ipv6_interface)
self.logger.debug("Ipv6 Interface: %s" % self.ipv6_interface)
def gen_auth(conf):
ask = True
if conf.login:
if conf.password:
yield conf.login, conf.password
ask = False
else:
yield conf.login, getpass.getpass()
while ask:
yield raw_input('SlapOS Master Login: '), getpass.getpass()
def register(config):
def do_register(conf):
"""Register new computer on SlapOS Master and generate slapos.cfg"""
# Get User identification and check them
if config.login == None :
while True :
print ("Please enter your SlapOS Master login")
user_id = get_login()
if check_login(user_id, config.master_url_web):
for login, password in gen_auth(conf):
if check_credentials(conf.master_url_web, login, password):
break
config.logger.warning ("Wrong login/password")
conf.logger.warning('Wrong login/password')
else:
if config.password == None :
config.password = getpass()
user_id = base64.encodestring('%s:%s' % (config.login, config.password))[:-1]
if not check_login(user_id, config.master_url_web):
config.logger.error ("Wrong login/password")
return 1
# Get source code of page having certificate and key
certificate_key = get_certificates(user_id, config.node_name, config.master_url_web)
certificate_key = get_certificates(conf.master_url_web, conf.node_name, login, password, conf.logger)
# Parse certificate and key and get computer id
certificate_key = parse_certificates(certificate_key)
certificate = certificate_key[0]
key = certificate_key[1]
certificate, key = parse_certificates(certificate_key)
COMP = get_computer_name(certificate)
# Getting configuration parameters
slapos_configuration = '/etc/opt/slapos/'
config.COMPConfig(slapos_configuration=slapos_configuration,
conf.COMPConfig(slapos_configuration='/etc/opt/slapos/',
computer_id=COMP,
certificate = certificate,
key = key
)
certificate=certificate,
key=key)
# Save former configuration
if not config.dry_run:
save_former_config(config)
if not conf.dry_run:
save_former_config(conf)
# Prepare Slapos Configuration
slapconfig(config)
slapconfig(conf)
print "Node has successfully been configured as %s." % COMP
return 0
def main():
"Run default configuration."
usage = "usage: slapos node %s NODE_NAME [options] " % sys.argv[0]
try:
# Parse arguments
config = Config()
config.setConfig(*Parser(usage=usage).check_args())
return_code = register(config)
except UsageError, err:
print >> sys.stderr, err.msg
print >> sys.stderr, "For help use --help"
return_code = 16
except ExecError, err:
print >> sys.stderr, err.msg
return_code = 16
except SystemExit, err:
# Catch exception raise by optparse
return_code = err
sys.exit(return_code)
slapos/slap/slap.py
View file @ 5f0924ff
......@@ -34,8 +34,6 @@ __all__ = ["slap", "ComputerPartition", "Computer", "SoftwareRelease",
"Supply", "OpenOrder", "NotFoundError", "Unauthorized",
"ResourceNotReady", "ServerError"]
from interface import slap as interface
from xml_marshaller import xml_marshaller
import httplib
import logging
import socket
......@@ -43,9 +41,16 @@ import ssl
import traceback
import urllib
import urlparse
import zope.interface
from interface import slap as interface
from xml_marshaller import xml_marshaller
fallback_logger = logging.getLogger(__name__)
fallback_handler = logging.StreamHandler()
fallback_logger.setLevel(logging.INFO)
fallback_logger.addHandler(fallback_handler)
log = logging.getLogger(__name__)
DEFAULT_SOFTWARE_TYPE = 'RootSoftwareInstance'
......@@ -144,7 +149,7 @@ class SoftwareRelease(SlapDocument):
else:
return self._software_release
def error(self, error_log):
def error(self, error_log, logger=None):
try:
# Does not follow interface
self._connection_helper.POST('/softwareReleaseError', {
......@@ -152,8 +157,7 @@ class SoftwareRelease(SlapDocument):
'computer_id' : self.getComputerId(),
'error_log': error_log})
except Exception:
exception = traceback.format_exc()
log.error(exception)
(logger or fallback_logger).error(traceback.format_exc())
def available(self):
self._connection_helper.POST('/availableSoftwareRelease', {
......@@ -417,15 +421,14 @@ class ComputerPartition(SlapRequester):
'computer_partition_id': self.getId(),
})
def error(self, error_log):
def error(self, error_log, logger=None):
try:
self._connection_helper.POST('/softwareInstanceError', {
'computer_id': self._computer_id,
'computer_partition_id': self.getId(),
'error_log': error_log})
except Exception:
exception = traceback.format_exc()
log.error(exception)
(logger or fallback_logger).error(traceback.format_exc())
def bang(self, message):
self._connection_helper.POST('/softwareInstanceBang', {
......@@ -660,7 +663,7 @@ class slap:
returns SoftwareRelease class object
"""
return SoftwareRelease(software_release=software_release,
connection_helper=self._connection_helper,
connection_helper=self._connection_helper
)
def registerComputer(self, computer_guid):
......
slapos/tests/entry.py
View file @ 5f0924ff
......@@ -28,7 +28,7 @@
import os
import shutil
import slapos.entry as entry
from slapos.cli_legacy import entry
import sys
import tempfile
import unittest
......@@ -126,15 +126,15 @@ class TestCall (BasicMixin, unittest.TestCase):
options = ["--logfile /opt/slapos/logfile",
"--pidfile /opt/slapos/pidfile"]
config = '/etc/opt/slapos/slapos.cfg'
config_path = '/etc/opt/slapos/slapos.cfg'
try:
entry.call(fun, config=config, option=options)
entry.call(fun, config_path=config_path, option=options)
except SystemExit, e:
self.assertEqual(e[0], 0)
self.assertNotEqual(original_sysargv, sys.argv)
for x in options:
self.assertTrue(x in " ".join(sys.argv))
self.assertEqual(config, sys.argv[1])
self.assertEqual(config_path, sys.argv[1])
def test_config_and_missing_option_are_added(self):
"""
......@@ -151,16 +151,16 @@ class TestCall (BasicMixin, unittest.TestCase):
return 0
options = [default_present_option, missing_option]
config = '/etc/opt/slapos/slapos.cfg'
config_path = '/etc/opt/slapos/slapos.cfg'
try:
entry.call(fun, config=config, option=options)
entry.call(fun, config_path=config_path, option=options)
except SystemExit, e:
self.assertEqual(e[0], 0)
self.assertNotEqual(original_sysargv, sys.argv)
for x in (missing_option, present_option):
self.assertTrue(x in " ".join(sys.argv))
self.assertFalse(default_present_option in " ".join(sys.argv))
self.assertEqual(config, sys.argv[1])
self.assertEqual(config_path, sys.argv[1])
def test_present_config_and_option_are_not_added(self):
"""
......@@ -178,9 +178,9 @@ class TestCall (BasicMixin, unittest.TestCase):
return 0
options = [default_present_option]
config = '/etc/opt/slapos/slapos.cfg'
config_path = '/etc/opt/slapos/slapos.cfg'
try:
entry.call(fun, config=config, option=options)
entry.call(fun, config_path=config_path, option=options)
except SystemExit, e:
self.assertEqual(e[0], 0)
......
slapos/tests/slap.py
View file @ 5f0924ff
......@@ -24,12 +24,15 @@
# Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.
#
##############################################################################
import xml_marshaller
import httplib
import os
import unittest
import urlparse
import httplib
import slapos.slap
import os
import xml_marshaller
class UndefinedYetException(Exception):
"""To catch exceptions which are not yet defined"""
......
slapos/tests/slapformat.py
View file @ 5f0924ff
......@@ -244,7 +244,9 @@ class TestComputer(SlapformatMixin):
@unittest.skip("Not implemented")
def test_construct_empty_prepared(self):
computer = slapos.format.Computer('computer',
interface=slapos.format.Interface('bridge', '127.0.0.1/16'))
interface=slapos.format.Interface(logger=self.test_result,
name='bridge',
ipv4_local_network='127.0.0.1/16'))
computer.instance_root = '/instance_root'
computer.software_root = '/software_root'
computer.construct()
......@@ -263,7 +265,9 @@ class TestComputer(SlapformatMixin):
def test_construct_empty_prepared_no_alter_user(self):
computer = slapos.format.Computer('computer',
interface=slapos.format.Interface('bridge', '127.0.0.1/16'))
interface=slapos.format.Interface(logger=self.test_result,
name='bridge',
ipv4_local_network='127.0.0.1/16'))
computer.instance_root = '/instance_root'
computer.software_root = '/software_root'
computer.construct(alter_user=False)
......@@ -279,7 +283,9 @@ class TestComputer(SlapformatMixin):
@unittest.skip("Not implemented")
def test_construct_empty_prepared_no_alter_network(self):
computer = slapos.format.Computer('computer',
interface=slapos.format.Interface('bridge', '127.0.0.1/16'))
interface=slapos.format.Interface(logger=self.test_result,
name='bridge',
ipv4_local_network='127.0.0.1/16'))
computer.instance_root = '/instance_root'
computer.software_root = '/software_root'
computer.construct(alter_network=False)
......@@ -298,7 +304,9 @@ class TestComputer(SlapformatMixin):
def test_construct_empty_prepared_no_alter_network_user(self):
computer = slapos.format.Computer('computer',
interface=slapos.format.Interface('bridge', '127.0.0.1/16'))
interface=slapos.format.Interface(logger=self.test_result,
name='bridge',
ipv4_local_network='127.0.0.1/16'))
computer.instance_root = '/instance_root'
computer.software_root = '/software_root'
computer.construct(alter_network=False, alter_user=False)
......@@ -315,7 +323,9 @@ class TestComputer(SlapformatMixin):
@unittest.skip("Not implemented")
def test_construct_prepared(self):
computer = slapos.format.Computer('computer',
interface=slapos.format.Interface('bridge', '127.0.0.1/16'))
interface=slapos.format.Interface(logger=self.test_result,
name='bridge',
ipv4_local_network='127.0.0.1/16'))
computer.instance_root = '/instance_root'
computer.software_root = '/software_root'
partition = slapos.format.Partition('partition', '/part_path',
......@@ -359,7 +369,9 @@ class TestComputer(SlapformatMixin):
def test_construct_prepared_no_alter_user(self):
computer = slapos.format.Computer('computer',
interface=slapos.format.Interface('bridge', '127.0.0.1/16'))
interface=slapos.format.Interface(logger=self.test_result,
name='bridge',
ipv4_local_network='127.0.0.1/16'))
computer.instance_root = '/instance_root'
computer.software_root = '/software_root'
partition = slapos.format.Partition('partition', '/part_path',
......@@ -400,7 +412,9 @@ class TestComputer(SlapformatMixin):
@unittest.skip("Not implemented")
def test_construct_prepared_no_alter_network(self):
computer = slapos.format.Computer('computer',
interface=slapos.format.Interface('bridge', '127.0.0.1/16'))
interface=slapos.format.Interface(logger=self.test_result,
name='bridge',
ipv4_local_network='127.0.0.1/16'))
computer.instance_root = '/instance_root'
computer.software_root = '/software_root'
partition = slapos.format.Partition('partition', '/part_path',
......@@ -440,7 +454,9 @@ class TestComputer(SlapformatMixin):
def test_construct_prepared_no_alter_network_user(self):
computer = slapos.format.Computer('computer',
interface=slapos.format.Interface('bridge', '127.0.0.1/16'))
interface=slapos.format.Interface(logger=self.test_result,
name='bridge',
ipv4_local_network='127.0.0.1/16'))
computer.instance_root = '/instance_root'
computer.software_root = '/software_root'
partition = slapos.format.Partition('partition', '/part_path',
......
slapos/tests/slapgrid.py
View file @ 5f0924ff
......@@ -25,25 +25,31 @@
#
##############################################################################
from slapos.grid import slapgrid
import httplib
import logging
import os
from random import random
import random
import shutil
import signal
import slapos.slap.slap
import slapos.grid.utils
from slapos.grid.watchdog import Watchdog, getWatchdogID
import socket
import sys
import tempfile
import textwrap
import time
import unittest
import unittest2
import urlparse
import xml_marshaller
import slapos.slap.slap
import slapos.grid.utils
from slapos.grid import slapgrid
from slapos.cli_legacy.slapgrid import parseArgumentTupleAndReturnSlapgridObject
from slapos.grid.utils import md5digest
from slapos.grid.watchdog import Watchdog, getWatchdogID
dummylogger = logging.getLogger()
WATCHDOG_TEMPLATE = """#!%(python_path)s -S
......@@ -112,10 +118,15 @@ class BasicMixin:
'supervisord')
self.usage_report_periodicity = 1
self.buildout = None
self.grid = slapgrid.Slapgrid(self.software_root, self.instance_root,
self.master_url, self.computer_id, self.supervisord_socket,
self.grid = slapgrid.Slapgrid(self.software_root,
self.instance_root,
self.master_url,
self.computer_id,
self.supervisord_socket,
self.supervisord_configuration_path,
self.buildout, develop=develop)
self.buildout,
develop=develop,
logger=logging.getLogger())
# monkey patch buildout bootstrap
def dummy(*args, **kw):
pass
......@@ -142,7 +153,7 @@ class BasicMixin:
shutil.rmtree(self._tempdir, True)
class TestBasicSlapgridCP(BasicMixin, unittest.TestCase):
class TestBasicSlapgridCP(BasicMixin, unittest2.TestCase):
def test_no_software_root(self):
self.assertRaises(OSError, self.grid.processComputerPartitionList)
......@@ -308,7 +319,7 @@ class ComputerForTest:
return (200, {}, '')
elif method == 'POST' and 'url' in parsed_qs:
# XXX hardcoded to first sofwtare release!
# XXX hardcoded to first software release!
software = self.software_list[0]
software.sequence.append(parsed_url.path)
if parsed_url.path == 'buildingSoftwareRelease':
......@@ -381,11 +392,11 @@ class InstanceForTest:
os.mkdir(certificate_repository_path)
self.cert_file = os.path.join(certificate_repository_path,
"%s.crt" % self.name)
self.certificate = str(random())
self.certificate = str(random.random())
open(self.cert_file, 'w').write(self.certificate)
self.key_file = os.path.join(certificate_repository_path,
"%s.key" % self.name)
self.key = str(random())
self.key = str(random.random())
open(self.key_file, 'w').write(self.key)
class SoftwareForTest:
......@@ -400,8 +411,7 @@ class SoftwareForTest:
self.software_root = software_root
self.name = 'http://sr%s/' % name
self.sequence = []
self.software_hash = \
slapos.grid.utils.getSoftwareUrlHash(self.name)
self.software_hash = md5digest(self.name)
self.srdir = os.path.join(self.software_root, self.software_hash)
self.requested_state = 'available'
os.mkdir(self.srdir)
......@@ -442,7 +452,7 @@ touch worked"""):
class TestSlapgridCPWithMaster(MasterMixin, unittest.TestCase):
class TestSlapgridCPWithMaster(MasterMixin, unittest2.TestCase):
def test_nothing_to_do(self):
......@@ -736,7 +746,7 @@ exit 1
self.assertEqual('stopped', instance.state)
class TestSlapgridCPWithMasterWatchdog(MasterMixin, unittest.TestCase):
class TestSlapgridCPWithMasterWatchdog(MasterMixin, unittest2.TestCase):
def setUp(self):
MasterMixin.setUp(self)
......@@ -911,7 +921,7 @@ class TestSlapgridCPWithMasterWatchdog(MasterMixin, unittest.TestCase):
self.assertEqual(computer.sequence,[])
class TestSlapgridCPPartitionProcessing (MasterMixin, unittest.TestCase):
class TestSlapgridCPPartitionProcessing (MasterMixin, unittest2.TestCase):
def test_partition_timestamp(self):
computer = ComputerForTest(self.software_root,self.instance_root)
......@@ -1298,7 +1308,7 @@ echo %s; echo %s; exit 42""" % (line1, line2))
self.assertTrue("Failed to run buildout" in instance.error_log)
class TestSlapgridUsageReport(MasterMixin, unittest.TestCase):
class TestSlapgridUsageReport(MasterMixin, unittest2.TestCase):
"""
Test suite about slapgrid-ur
"""
......@@ -1500,7 +1510,7 @@ class TestSlapgridUsageReport(MasterMixin, unittest.TestCase):
class TestSlapgridSoftwareRelease(MasterMixin, unittest.TestCase):
class TestSlapgridSoftwareRelease(MasterMixin, unittest2.TestCase):
def test_one_software_buildout_fail_is_correctly_logged(self):
"""
1. We set up a software using a corrupted buildout
......@@ -1521,7 +1531,7 @@ echo %s; echo %s; exit 42""" % (line1, line2))
self.assertTrue(line2 in software.error_log)
self.assertTrue("Failed to run buildout" in software.error_log)
class SlapgridInitialization(unittest.TestCase):
class SlapgridInitialization(unittest2.TestCase):
"""
"Abstract" class setting setup and teardown for TestSlapgridArgumentTuple
and TestSlapgridConfigurationFile.
......@@ -1571,7 +1581,7 @@ class TestSlapgridArgumentTuple(SlapgridInitialization):
"""
Raises if the argument list if empty and without configuration file.
"""
parser = slapgrid.parseArgumentTupleAndReturnSlapgridObject
parser = parseArgumentTupleAndReturnSlapgridObject
# XXX: SystemExit is too generic exception, it is only known that
# something is wrong
self.assertRaises(SystemExit, parser, *())
......@@ -1581,7 +1591,7 @@ class TestSlapgridArgumentTuple(SlapgridInitialization):
Check if we can have the slapgrid object returned with the minimum
arguments.
"""
parser = slapgrid.parseArgumentTupleAndReturnSlapgridObject
parser = parseArgumentTupleAndReturnSlapgridObject
return_list = parser(*self.default_arg_tuple)
self.assertEquals(2, len(return_list))
......@@ -1589,19 +1599,19 @@ class TestSlapgridArgumentTuple(SlapgridInitialization):
"""
Raises if the signature_private_key_file does not exists.
"""
parser = slapgrid.parseArgumentTupleAndReturnSlapgridObject
parser = parseArgumentTupleAndReturnSlapgridObject
argument_tuple = ("--signature_private_key_file", "/non/exists/path") + \
self.default_arg_tuple
# XXX: SystemExit is too generic exception, it is only known that
# something is wrong
self.assertRaises(SystemExit, parser, *argument_tuple)
self.assertRaisesRegexp(RuntimeError,
"File '/non/exists/path' does not exist.",
parser, *argument_tuple)
def test_signature_private_key_file(self):
"""
Check if the signature private key argument value is available on
slapgrid object.
"""
parser = slapgrid.parseArgumentTupleAndReturnSlapgridObject
parser = parseArgumentTupleAndReturnSlapgridObject
argument_tuple = ("--signature_private_key_file",
self.signature_key_file_descriptor.name) + \
self.default_arg_tuple
......@@ -1613,7 +1623,7 @@ class TestSlapgridArgumentTuple(SlapgridInitialization):
"""
Check if giving --all triggers "develop" option.
"""
parser = slapgrid.parseArgumentTupleAndReturnSlapgridObject
parser = parseArgumentTupleAndReturnSlapgridObject
argument_tuple = ("--all",) + self.default_arg_tuple
slapgrid_object = parser(*argument_tuple)[0]
self.assertTrue(slapgrid_object.develop)
......@@ -1623,7 +1633,7 @@ class TestSlapgridArgumentTuple(SlapgridInitialization):
Check if not giving --all neither --develop triggers "develop"
option to be False.
"""
parser = slapgrid.parseArgumentTupleAndReturnSlapgridObject
parser = parseArgumentTupleAndReturnSlapgridObject
argument_tuple = self.default_arg_tuple
slapgrid_object = parser(*argument_tuple)[0]
self.assertFalse(slapgrid_object.develop)
......@@ -1633,7 +1643,7 @@ class TestSlapgridArgumentTuple(SlapgridInitialization):
Check if not giving --maximum-periodicity triggers "force_periodicity"
option to be false.
"""
parser = slapgrid.parseArgumentTupleAndReturnSlapgridObject
parser = parseArgumentTupleAndReturnSlapgridObject
argument_tuple = self.default_arg_tuple
slapgrid_object = parser(*argument_tuple)[0]
self.assertFalse(slapgrid_object.force_periodicity)
......@@ -1642,7 +1652,7 @@ class TestSlapgridArgumentTuple(SlapgridInitialization):
"""
Check if giving --maximum-periodicity triggers "force_periodicity" option.
"""
parser = slapgrid.parseArgumentTupleAndReturnSlapgridObject
parser = parseArgumentTupleAndReturnSlapgridObject
argument_tuple = ("--maximum-periodicity","40") + self.default_arg_tuple
slapgrid_object = parser(*argument_tuple)[0]
self.assertTrue(slapgrid_object.force_periodicity)
......@@ -1666,7 +1676,7 @@ upload-to-binary-cache-url-blacklist =
http://2/bla
""" % dict(fake_file=self.fake_file_descriptor.name))
self.slapos_config_descriptor.seek(0)
slapgrid_object = slapgrid.parseArgumentTupleAndReturnSlapgridObject(
slapgrid_object = parseArgumentTupleAndReturnSlapgridObject(
*self.default_arg_tuple)[0]
self.assertEqual(
slapgrid_object.upload_to_binary_cache_url_blacklist,
......@@ -1694,7 +1704,7 @@ download-from-binary-cache-url-blacklist =
http://2/bla
""" % dict(fake_file=self.fake_file_descriptor.name))
self.slapos_config_descriptor.seek(0)
slapgrid_object = slapgrid.parseArgumentTupleAndReturnSlapgridObject(
slapgrid_object = parseArgumentTupleAndReturnSlapgridObject(
*self.default_arg_tuple)[0]
self.assertEqual(
slapgrid_object.upload_to_binary_cache_url_blacklist,
......@@ -1726,7 +1736,7 @@ download-from-binary-cache-url-blacklist =
http://4/bla
""" % dict(fake_file=self.fake_file_descriptor.name))
self.slapos_config_descriptor.seek(0)
slapgrid_object = slapgrid.parseArgumentTupleAndReturnSlapgridObject(
slapgrid_object = parseArgumentTupleAndReturnSlapgridObject(
*self.default_arg_tuple)[0]
self.assertEqual(
slapgrid_object.upload_to_binary_cache_url_blacklist,
......@@ -1755,7 +1765,7 @@ binary-cache-url-blacklist =
http://2/bla
""" % dict(fake_file=self.fake_file_descriptor.name))
self.slapos_config_descriptor.seek(0)
slapgrid_object = slapgrid.parseArgumentTupleAndReturnSlapgridObject(
slapgrid_object = parseArgumentTupleAndReturnSlapgridObject(
*self.default_arg_tuple)[0]
self.assertEqual(
slapgrid_object.upload_to_binary_cache_url_blacklist,
......@@ -1767,7 +1777,7 @@ binary-cache-url-blacklist =
)
class TestSlapgridCPWithMasterPromise(MasterMixin, unittest.TestCase):
class TestSlapgridCPWithMasterPromise(MasterMixin, unittest2.TestCase):
def test_one_failing_promise(self):
computer = ComputerForTest(self.software_root,self.instance_root)
instance = computer.instance_list[0]
......@@ -1822,7 +1832,7 @@ exit 127""" % {'worked_file': worked_file})
slapos.grid.slapgrid.SLAPGRID_PROMISE_FAIL)
self.assertTrue(os.path.isfile(worked_file))
self.assertEqual(instance.error_log, 'Error')
self.assertEqual(instance.error_log[-5:], 'Error')
self.assertTrue(instance.error)
self.assertIsNone(instance.state)
......
slapos/tests/slapobject.py
View file @ 5f0924ff
......@@ -25,12 +25,14 @@
#
##############################################################################
import logging
import os
import unittest
from slapos.grid import SlapObject
from slapos.grid import utils
from slapos.grid import networkcache
from slapos.tests.slapgrid import BasicMixin
import os
import unittest
class FakeCallAndRead:
......@@ -93,6 +95,7 @@ class TestSoftwareSlapObject(BasicMixin, unittest.TestCase):
url='http://example.com/software.cfg',
software_root=self.software_root,
buildout=self.buildout,
logger=logging.getLogger(),
signature_private_key_file='/signature/private/key_file',
upload_cache_url='http://example.com/uploadcache',
upload_dir_url='http://example.com/uploaddir',
......@@ -129,7 +132,8 @@ class TestSoftwareSlapObject(BasicMixin, unittest.TestCase):
software = SlapObject.Software(
url='http://example.com/software.cfg',
software_root=self.software_root,
buildout=self.buildout)
buildout=self.buildout,
logger=logging.getLogger())
software.install()
......@@ -161,6 +165,7 @@ class TestSoftwareSlapObject(BasicMixin, unittest.TestCase):
url='http://example.com/software.cfg',
software_root=self.software_root,
buildout=self.buildout,
logger=logging.getLogger(),
signature_private_key_file='/signature/private/key_file',
upload_cache_url='http://example.com/uploadcache',
upload_dir_url='http://example.com/uploaddir',
......@@ -192,6 +197,7 @@ class TestSoftwareSlapObject(BasicMixin, unittest.TestCase):
url='http://example.com/software.cfg',
software_root=self.software_root,
buildout=self.buildout,
logger=logging.getLogger(),
signature_private_key_file='/signature/private/key_file',
upload_cache_url='http://example.com/uploadcache',
upload_dir_url='http://example.com/uploaddir',
......
slapos/tests/slapproxy.py
View file @ 5f0924ff
......@@ -27,6 +27,7 @@
#
##############################################################################
import ConfigParser
import os
import logging
import shutil
......@@ -86,14 +87,16 @@ database_uri = %(tempdir)s/lib/proxy.db
"""
Set config for slapproxy and start it
"""
config = slapos.proxy.Config()
config.setConfig(*(ProxyOption(self.proxy_db),
self.slapos_cfg))
conf = slapos.proxy.ProxyConfig(logger=logging.getLogger())
configp = ConfigParser.SafeConfigParser()
configp.read(self.slapos_cfg)
conf.mergeConfig(ProxyOption(self.proxy_db), configp)
conf.setConfig()
views.app.config['TESTING'] = True
views.app.config['computer_id'] = self.computer_id
views.app.config['DATABASE_URI'] = self.proxy_db
views.app.config['HOST'] = config.host
views.app.config['port'] = config.port
views.app.config['HOST'] = conf.host
views.app.config['port'] = conf.port
self.app = views.app.test_client()
def add_free_partition (self, partition_amount):
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment
GitLab Nexedi Edition | About GitLab | About Nexedi | 沪ICP备2021021310号-2 | 沪ICP备2021021310号-7