Commit e26b498f authored by Thomas Gambier's avatar Thomas Gambier

cleanup: totally remove deprecated software/slaprunner

See merge request nexedi/slapos!1329
parent 686adec3
[buildout]
extends =
../../slaprunner/development.cfg
testsuite.cfg
parts += template-erp5testnode
[template-resilient-test]
filename = instance-resilient-test.cfg.jinja2
md5sum = e4b04aa6fd3413bc6ae38823abdcc8fa
# We have to use an extra level of indentation here because this is substituted
# during software buildout to generate instance buildout, but the
# slapos.recipe.template recipe doing the substitution does string replacements
# without knowledge of the buildout syntax, so we want the second line to be
# indented in the final generated instance buildout.
extra-context =
raw slapos_repository_url ${slapos.cookbook-repository:repository}
raw slapos_repository_branch ${slapos.cookbook-repository:branch}
[exporter-default-configuration]
# Define shorter interaction to speed up tests
backup_wait_time = 1
# Development profile of slaprunner. # Development profile of kvm.
# Exactly the same as software.cfg, but fetch the slapos.cookbook and # Exactly the same as software.cfg, but fetch the slapos.cookbook and
# slapos.toolbox from git repository instead of fetching stable version, # slapos.toolbox from git repository instead of fetching stable version,
# allowing to play with bleeding edge environment. # allowing to play with bleeding edge environment.
......
...@@ -12,7 +12,7 @@ The results of this test suite running on Nexedi ERP5 are published as `SlapOS.S ...@@ -12,7 +12,7 @@ The results of this test suite running on Nexedi ERP5 are published as `SlapOS.S
## Running test locally ## Running test locally
Here's an example session of how a developer could use this software release in Here's an example session of how a developer could use this software release in
slaprunner to develop a slapos profile, in the example `helloworld`, make Theia to develop a slapos profile, in the example `helloworld`, make
changes to the code, run tests and publish changes. changes to the code, run tests and publish changes.
```bash ```bash
......
...@@ -117,11 +117,6 @@ setup = ${slapos-repository:location}/software/restic-rest-server/test/ ...@@ -117,11 +117,6 @@ setup = ${slapos-repository:location}/software/restic-rest-server/test/
egg = slapos.test.seleniumserver egg = slapos.test.seleniumserver
setup = ${slapos-repository:location}/software/seleniumserver/test/ setup = ${slapos-repository:location}/software/seleniumserver/test/
[slapos.test.slaprunner-setup]
<= setup-develop-egg
egg = slapos.test.slaprunner
setup = ${slapos-repository:location}/software/slaprunner/test/
[slapos.test.metabase-setup] [slapos.test.metabase-setup]
<= setup-develop-egg <= setup-develop-egg
egg = slapos.test.metabase egg = slapos.test.metabase
...@@ -337,7 +332,6 @@ eggs += ...@@ -337,7 +332,6 @@ eggs +=
${slapos.test.restic_rest_server-setup:egg} ${slapos.test.restic_rest_server-setup:egg}
${slapos.test.seleniumserver-setup:egg} ${slapos.test.seleniumserver-setup:egg}
${slapos.test.slapos-master-setup:egg} ${slapos.test.slapos-master-setup:egg}
${slapos.test.slaprunner-setup:egg}
${slapos.test.theia-setup:egg} ${slapos.test.theia-setup:egg}
${slapos.test.turnserver-setup:egg} ${slapos.test.turnserver-setup:egg}
${slapos.test.upgrade_erp5-setup:egg} ${slapos.test.upgrade_erp5-setup:egg}
...@@ -428,7 +422,6 @@ tests = ...@@ -428,7 +422,6 @@ tests =
restic-rest-server ${slapos.test.restic_rest_server-setup:setup} restic-rest-server ${slapos.test.restic_rest_server-setup:setup}
seleniumserver ${slapos.test.seleniumserver-setup:setup} seleniumserver ${slapos.test.seleniumserver-setup:setup}
slapos-master ${slapos.test.slapos-master-setup:setup} slapos-master ${slapos.test.slapos-master-setup:setup}
slaprunner ${slapos.test.slaprunner-setup:setup}
theia ${slapos.test.theia-setup:setup} theia ${slapos.test.theia-setup:setup}
turnserver ${slapos.test.turnserver-setup:setup} turnserver ${slapos.test.turnserver-setup:setup}
upgrade_erp5 ${slapos.test.upgrade_erp5-setup:setup} upgrade_erp5 ${slapos.test.upgrade_erp5-setup:setup}
......
...@@ -10,7 +10,7 @@ The results of this test suite running on Nexedi ERP5 are published as ...@@ -10,7 +10,7 @@ The results of this test suite running on Nexedi ERP5 are published as
Here's an example session of how a developer could use this software release in Here's an example session of how a developer could use this software release in
slaprunner to develop a slapos egg, in the example `slapos.core`, to make Theia to develop a slapos egg, in the example `slapos.core`, to make
changes to the code, run tests and publish changes. changes to the code, run tests and publish changes.
```bash ```bash
......
slaprunner
==========
Introduction
------------
This software release is used to deploy Slaprunner instances.
Slaprunner is an all-in-one IDE used to develop and test profiles and recipes for SlapOS.
This software is only running in Python2 and has been deprecated in favor of theia.
See https://lab.nexedi.com/nexedi/slapos/tree/master/software/theia
- resilient sr: Cloned instances should not launch slapgrid-sr if it was not launched on export instance
- add test for parameter auto-deploy-instance
- Add download facility in file browser
# THIS IS NOT A BUILDOUT FILE, despite purposedly using a compatible syntax.
# The only allowed lines here are (regexes):
# - "^#" comments, copied verbatim
# - "^[" section beginings, copied verbatim
# - lines containing an "=" sign which must fit in the following categorie.
# - "^\s*filename\s*=\s*path\s*$" where "path" is relative to this file
# Copied verbatim.
# - "^\s*hashtype\s*=.*" where "hashtype" is one of the values supported
# by the re-generation script.
# Re-generated.
# - other lines are copied verbatim
# Substitution (${...:...}), extension ([buildout] extends = ...) and
# section inheritance (< = ...) are NOT supported (but you should really
# not need these here).
[template]
filename = instance.cfg
md5sum = 473325442cba8ee32642d1e5e3fa94f4
[template-runner]
filename = instance-runner.cfg
md5sum = 03d3774fcb01a8ec68a23de31fb346b8
[template-runner-import-script]
filename = template/runner-import.sh.jinja2
md5sum = f2e2493bc5da90a53f86e5bcf64d2d57
[instance-runner-import]
filename = instance-runner-import.cfg.in
md5sum = f4d71d97cf16ef379b41c4c22352727c
[instance-runner-export]
filename = instance-runner-export.cfg.in
md5sum = 5f57249108e61976d97465624cc5a2a1
[template-resilient]
filename = instance-resilient.cfg.jinja2
md5sum = bd0ad0b80d2b39189f9665c48f1b3830
[template_nginx_conf]
filename = nginx_conf.in
md5sum = a042e52e0594ec0486cff7dc896cb6ce
[template_httpd_conf]
filename = httpd_conf.in
md5sum = 35f5f5c6e183af6064398c88fa85de59
[template_launcher]
filename = launcher.in
md5sum = 7a2f5e25a818cb29964666ada7852a5c
[template-slapos-cfg]
filename = template/slapos.cfg.in
md5sum = 03322b8a0f42ea579d35981211359190
[template-slapformat-definition.cfg]
filename = template/slapformat-definition.cfg.in
md5sum = bbb767dee9730f5a816ce4341403d89b
[template-parameters]
filename = parameters.xml.in
md5sum = 0d7b2432aa809b859509ca1114accd8f
[template-bash-profile]
filename = template/bash_profile.in
md5sum = d0ff3c86c76168d17d27b47b93a9b7aa
[template-supervisord]
filename = template/supervisord.conf.in
md5sum = 5e6c84098440c6bc163898dcafca8c9b
[template-listener-slapgrid]
filename = template/listener_slapgrid.py.in
md5sum = 49d50410cf7467175a841eb7cd0d93d4
[monitor-check-webrunner-internal-instance]
filename = template/monitor-check-webrunner-internal-instances.py
md5sum = acaac32cf1bd45714272468a89f4f119
[template-resilient-software-release-information]
filename = template/resilient_software_release_information.py.in
md5sum = 869a3afbf3c2fff6c72602662c6a8730
[template-slapuser-script]
filename = template/slapos-slapuser-script.in
md5sum = 75aab99c995ca841f93fc77fc9116c37
[template-buildout-shared-part-list]
filename = template/buildout-shared-part-list.in
md5sum = f619c8c5897c4851442b7090c8509758
# Development profile of slaprunner.
# Exactly the same as software.cfg, but fetch the slapos.cookbook and
# slapos.toolbox from git repository instead of fetching stable version,
# allowing to play with bleeding edge environment.
[buildout]
extends = software.cfg
../../stack/slapos-dev.cfg
parts +=
# Development eggs
${:common-parts}
[versions]
slapos.cookbook =
slapos.core =
slapos.toolbox =
PidFile "{{ parameters.path_pid }}"
ServerName example.com
ServerAdmin someone@email
<IfDefine !HTTPDPort>
Listen [{{ parameters.global_ip }}]:{{ parameters.global_port }}
Define HTTPDPort
</IfDefine>
LoadModule unixd_module modules/mod_unixd.so
LoadModule access_compat_module modules/mod_access_compat.so
LoadModule auth_basic_module modules/mod_auth_basic.so
LoadModule authz_core_module modules/mod_authz_core.so
LoadModule authz_user_module modules/mod_authz_user.so
LoadModule authz_host_module modules/mod_authz_host.so
LoadModule authn_core_module modules/mod_authn_core.so
LoadModule authn_file_module modules/mod_authn_file.so
LoadModule mime_module modules/mod_mime.so
LoadModule ssl_module modules/mod_ssl.so
LoadModule alias_module modules/mod_alias.so
LoadModule env_module modules/mod_env.so
LoadModule rewrite_module modules/mod_rewrite.so
LoadModule headers_module modules/mod_headers.so
LoadModule log_config_module modules/mod_log_config.so
LoadModule dav_module modules/mod_dav.so
LoadModule dav_fs_module modules/mod_dav_fs.so
LoadModule cache_module modules/mod_cache.so
LoadModule file_cache_module modules/mod_file_cache.so
LoadModule setenvif_module modules/mod_setenvif.so
LoadModule dir_module modules/mod_dir.so
LoadModule autoindex_module modules/mod_autoindex.so
ErrorLog "{{ parameters.path_error_log }}"
LogFormat "%h %l %u %t \"%r\" %>s %b" common
CustomLog "{{ parameters.path_access_log }}" common
# SSL Configuration
Define SSLConfigured
SSLCertificateFile {{ parameters.cert_file }}
SSLCertificateKeyFile {{ parameters.key_file }}
SSLRandomSeed startup builtin
SSLRandomSeed connect builtin
SSLRandomSeed startup /dev/urandom 256
SSLRandomSeed connect builtin
SSLProtocol all -SSLv2 -SSLv3
SSLCipherSuite ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+3DES:HIGH:!aNULL:!MD5
SSLHonorCipherOrder on
SSLEngine On
Include {{ parameters.httpd_cors_file }}
DocumentRoot {{ parameters.runner_home }}/public
# Directory protection
<Directory />
Options FollowSymLinks
AllowOverride None
Require all denied
</Directory>
Alias /public {{ parameters.runner_home }}/public
<Directory {{ parameters.runner_home }}/public>
<Limit GET HEAD OPTIONS REPORT PROPFIND>
Order Allow,Deny
Allow from all
AllowOverride All
Satisfy Any
Options Indexes FollowSymLinks
DirectoryIndex index.html
</Limit>
<Files .htaccess>
order allow,deny
deny from all
</Files>
</Directory>
DavLockDB {{ parameters.dav_lock }}
Alias /share {{ parameters.runner_home }}
<Directory {{ parameters.runner_home }}>
DirectoryIndex disabled
DAV On
Options Indexes FollowSymLinks
AuthType Basic
AuthName "Webrunner Dav"
AuthUserFile "{{ parameters.htpasswd_file }}"
# Prevent using the web browser cache if requesting the same document
# from different domains or with different users
Header set Cache-Control "private, max-age=0, must-revalidate"
Header set Vary "Origin,Cookie,Authorization"
<LimitExcept OPTIONS>
Require valid-user
</LimitExcept>
</Directory>
# vim: set ft=cfg:
# Default parameters
# NOTE: in case of resilient instance, auto-deploy-instance
# set to false (else it could be dangerous)
{% do slapparameter_dict.__setitem__('auto-deploy', slapparameter_dict.get('auto-deploy', 'true')) %}
{% do slapparameter_dict.__setitem__('auto-deploy-instance', slapparameter_dict.get('auto-deploy-instance', 'false')) %}
{% do slapparameter_dict.__setitem__('instance-type', 'resilient') %}
# let decide the number of clones we want. We can't have more than 2 clones
{% set number_of_instances = slapparameter_dict.get('resilient-clone-number', 1)|int %}
{% if number_of_instances > 2 %}
{% set number_of_instances = 2 %}
{% endif %}
{% set slaprunner_return = ['init-user', 'init-password', 'url', 'ssh-public-key', 'resilient-ssh-url', 'notification-id', 'ip', 'backend-url', 'url', 'ssh-url', 'ssh-command', 'webdav-url', 'public-url'] -%}
{% set monitor_return = ['monitor-base-url', 'monitor-url', 'monitor-user', 'monitor-password'] -%}
{% set monitor_parameter = {'monitor-cors-domains': slapparameter_dict.pop('monitor-cors-domains', "monitor.app.officejs.com")} -%}
{% set monitor_dict = {'parameter': monitor_parameter, 'return': monitor_return, 'set-monitor-url': True} -%}
{% set monitor_interface_url = slapparameter_dict.pop('monitor-interface-url', 'https://monitor.app.officejs.com') -%}
{% import 'parts' as parts %}
{% import 'replicated' as replicated with context %}
[buildout]
eggs-directory = {{ eggs_directory }}
develop-eggs-directory = {{ develop_eggs_directory }}
offline = true
extends =
{{ monitor_template }}
# += because we need to take up parts (like instance-custom, slapmonitor etc) from the profile we extended
parts +=
publish-early
{{ parts.replicate("runner", number_of_instances + 1) }}
publish-connection-information
[monitor-htpasswd]
recipe = slapos.cookbook:generate.password
storage-path = ${directory:etc}/.monitor_user
bytes = 8
username = admin
[publish-early]
recipe = slapos.cookbook:publish-early
-init =
init-password monitor-htpasswd:passwd
{% do monitor_parameter.__setitem__('monitor-username', slapparameter_dict.get('monitor-username', 'admin'))%}
{% do monitor_parameter.__setitem__('monitor-password', slapparameter_dict.get('monitor-password', '${publish-early:init-password}'))%}
{{ replicated.replicate("runner", number_of_instances + 1, "runner-export", "runner-import", slapparameter_dict=slapparameter_dict, monitor_parameter_dict=monitor_dict) }}
[directory]
recipe = slapos.cookbook:mkdirectory
etc = ${buildout:directory}/etc
# Bubble up the parameters
[request-runner]
{% if slapparameter_dict.get('custom-frontend-backend-url') -%}
{% do slaprunner_return.append('custom-frontend-url') -%}
{% endif -%}
return = {{ slaprunner_return | join(' ')}} {{ monitor_return | join(' ') }}
[publish-connection-information]
recipe = slapos.cookbook:publish
backend-url = ${request-runner:connection-backend-url}
url = ${request-runner:connection-url}
init-user = ${request-runner:connection-init-user}
init-password = ${publish-early:init-password}
ssh-command = ${request-runner:connection-ssh-command}
webdav-url = ${request-runner:connection-webdav-url}
public-url = ${request-runner:connection-public-url}
{% if slapparameter_dict.get('custom-frontend-backend-url') -%}
custom-frontend-url = ${request-runner:connection-custom-frontend-url}
{% endif %}
monitor-base-url = ${request-runner:connection-monitor-base-url}
monitor-setup-url = {{ monitor_interface_url }}/#page=settings_configurator&url=${request-runner:connection-monitor-url}&username=${request-runner:connection-monitor-user}&password=${request-runner:connection-monitor-password}
[slap-parameter]
# Default parameters for distributed deployment
# I.e state "backup1 of maria should go there, ..."
# XXX-Cedric: Hardcoded number of backups. Should be dynamically generated.
{% for nb in range(1,number_of_instances+1) %}
runner{{nb}}-computer-guid =
pbs-runner{{nb}}-computer-guid =
{% endfor %}
# XXX-Cedric: Hardcoded parameters. Should be dynamically generated.
domain =
authorized-key =
instance-amount = 10
debug = false
resilient-clone-number = 1
[buildout]
extends = {{ template_runner_path }}
{{ pbsready_export_template_path }}
# parts from {{ template_runner_path }} are overriden by
# {{ pbsready_export_template_path }} so we reinclude them here
parts += ${:common-runner-parts}
[directory]
recipe = slapos.cookbook:mkdirectory
# XXX - keep srv path with slash at the end.
srv = ${:home}/srv/
[proxy-free-port]
recipe = slapos.cookbook:free_port
minimum = 49980
maximum = 49989
ip = ${slap-network-information:local-ipv4}
[runner-free-port]
recipe = slapos.cookbook:free_port
minimum = 50005
maximum = 50014
ip = ${slap-network-information:global-ipv6}
[slaprunner]
proxy_port = ${proxy-free-port:port}
runner_port = ${runner-free-port:port}
[supervisord-free-port]
recipe = slapos.cookbook:free_port
minimum = 39986
maximum = 39995
ip = ${slaprunner:ipv4}
[supervisord]
port = ${supervisord-free-port:port}
[exporter-raw]
recipe = slapos.cookbook:wrapper
wrapper-path = ${directory:bin}/${slap-parameter:namebase}-exporter-raw
command-line = {{ software_release_bin }}/${slap-parameter:namebase}-exporter --srv-path ${directory:srv} --etc-path ${directory:etc} --backup-path ${directory:backup} --backup-wait-time {{ backup_wait_time }}
environment =
PATH={{ rsync_bin_folder }}:/bin:/usr/bin
[exporter]
recipe = slapos.cookbook:wrapper
command-line = {{ bash_executable_location }} -c "exec ${exporter-raw:wrapper-path} > >(tee -ai "${directory:log}/resilient.log") 2>&1"
wrapper-path = ${directory:bin}/${slap-parameter:namebase}-exporter
# wrapper parameter is needed by resilience stack
wrapper = ${:wrapper-path}
[monitor-httpd-free-port]
recipe = slapos.cookbook:free_port
minimum = 8437
maximum = 8446
ip = ${slap-network-information:global-ipv6}
# XXX Redefine Monitor parameters for runner-export
[monitor-instance-parameter]
monitor-httpd-port = ${monitor-httpd-free-port:port}
{% if slapparameter_dict.get('name', '') -%}
monitor-title = {{ slapparameter_dict['name'] }}
{% endif -%}
cors-domains = {{ slapparameter_dict.get('monitor-cors-domains', 'monitor.app.officejs.com') }}
{% if slapparameter_dict.get('monitor-username', '') -%}
username = {{ slapparameter_dict['monitor-username'] }}
{% endif -%}
{% if slapparameter_dict.get('monitor-password', '') -%}
password = {{ slapparameter_dict['monitor-password'] }}
{% endif -%}
{% if slapparameter_dict.get('monitor-url-list', '') -%}
monitor-url-list = {{ slapparameter_dict['monitor-url-list'] }}
{% endif -%}
instance-configuration =
httpdcors cors-domain ${slaprunner-httpd-cors:location} ${httpd-graceful-wrapper:output}
configuration-file-path = ${buildout:directory}/knowledge0.cfg
[monitor-conf-parameters]
private-path-list +=
$${logrotate-directory:logrotate-backup}
# Extends publish section with resilient parameters
[publish-connection-information]
<= resilient-publish-connection-parameter
[monitor-check-resilient-feed-file]
recipe = slapos.recipe.template:jinja2
url = {{ monitor_check_resilient_feed_template_path }}
output = ${monitor-directory:bin}/check-create-resilient-feed-files
context =
key input_feed_directory directory:notifier-feeds
key monitor_feed_directory monitor-directory:public
raw base_url http://[${notifier:host}]:${notifier:port}/get/
raw python_executable ${buildout:executable}
[buildout]
extends = {{ template_runner_path }}
{{ pbsready_import_template_path }}
parts +=
nginx_conf
nginx-launcher
certificate-authority
ca-nginx
certificate-authority-service
ca-nginx-service
gunicorn-launcher
gunicorn-graceful
slaprunner-promise
slaprunner-supervisord-wrapper
runner-sshd-add-authorized-key
runner-sshd-promise
runner-sshd-service
runtestsuite
shellinabox
shellinabox-service
slapos-cfg
cron-entry-prepare-software
deploy-instance-parameters
instance-software-type
bash-profile
supervisord-wrapper
importer-consistency-promise
software-release-deployment-promise
template-slapuser-script
resilient-software-release-information
monitor-base
[directory]
recipe = slapos.cookbook:mkdirectory
# XXX - keep srv path with slash at the end.
srv = ${:home}/srv/
# For the needs of importer, we run the full slaprunner
# In case both exporter and importer (aka main instance and clone instance)
# run with the same IP (usually for testing purposes),
# run slaprunner using different ports.
[proxy-free-port]
recipe = slapos.cookbook:free_port
minimum = 49990
maximum = 49999
ip = ${slap-network-information:local-ipv4}
[runner-free-port]
recipe = slapos.cookbook:free_port
minimum = 50015
maximum = 50024
ip = ${slap-network-information:global-ipv6}
[slaprunner]
proxy_port = ${proxy-free-port:port}
runner_port = ${runner-free-port:port}
[supervisord-free-port]
recipe = slapos.cookbook:free_port
minimum = 39996
maximum = 40005
ip = ${slaprunner:ipv4}
[supervisord]
port = ${supervisord-free-port:port}
[runner-sshd-port]
minimum = 22232
maximum = 22241
# Deactivate the call to prepareSoftware, and let the importer script
# handle the build&run of the instance.
[cron-entry-prepare-software]
recipe =
[importer]
recipe = slapos.recipe.template:jinja2
url = {{ importer_script_path }}
output = ${directory:bin}/${slap-parameter:namebase}-importer
# backward compatibility for resilient stack
wrapper = ${:output}
restore-exit-code-file = ${directory:srv}/${:restore-exit-code-file-basename}
restore-exit-code-file-basename = importer-exit-code-file
restore-error-message-file = ${directory:srv}/${:restore-error-message-file-basename}
restore-error-message-file-basename = importer-error-message-file
resilient-log-basename = resilient.log
context =
import sys sys
import easy_install zc.buildout.easy_install
key backend_url slaprunner:access-url
key ipv4 slaprunner:ipv4
key ipv6 slaprunner:ipv6
key proxy_port slaprunner:proxy_port
key instance_folder slaprunner:instance_root
section directory directory
section supervisord supervisord
raw output_log_file ${directory:log}/${:resilient-log-basename}
raw shell_binary {{ bash_executable_location }}
raw sqlite3_binary {{ sqlite3_executable_location }}
raw rsync_binary {{ rsync_executable_location }}
raw restore_exit_code_file ${:restore-exit-code-file}
raw restore_error_message_file ${:restore-error-message-file}
[importer-consistency-promise-bin]
# Test that the importer script and "after-import" subscripts
# are not older than 2 days (1 day + some slack), and have succeeded
recipe = collective.recipe.template
input = inline: #!/bin/sh
EXIT_CODE_FILE="${importer:restore-exit-code-file}"
RECENT_EXIT_CODE_FILE=$(find ${directory:srv} -maxdepth 1 -name "${importer:restore-exit-code-file-basename}" -mtime -2)
RESILIENT_LOG_URL=${monitor-publish-parameters:monitor-base-url}/share/private/log/${importer:resilient-log-basename}
if [ ! -f "$EXIT_CODE_FILE" ]; then
exit 0;
else
if [ -z "$RECENT_EXIT_CODE_FILE" ]; then
echo "Consistency check is too old.";
exit 1;
else
if [ -f "${importer:restore-error-message-file}" ]; then
cat ${importer:restore-error-message-file}
fi
echo "More information can be found here : $RESILIENT_LOG_URL";
exit $(cat $EXIT_CODE_FILE);
fi
fi
exit 1; # Something else went wrong
output = ${directory:bin}/importer-consistency-promise
mode = 755
[importer-consistency-promise]
<= monitor-promise-base
promise = check_command_execute
name = importer-consistency-promise.py
config-command = ${importer-consistency-promise-bin:output}
[software-release-deployment-bin]
recipe = collective.recipe.template
input = inline: #!/bin/sh
PROJECT_FILE=$(find "${directory:etc}" -maxdepth 1 -name .project)
if [ -z "$PROJECT_FILE" ]; then
exit 0;
else
INSTALLATION_FILE=$(find "${runnerdirectory:software-root}" -mindepth 2 -maxdepth 2 -name .completed)
if [ -n "$INSTALLATION_FILE" ]; then
exit 0;
else
exit 1;
fi
fi
exit 1
output = ${directory:bin}/software-release-deployment-promise
mode = 755
[software-release-deployment-promise]
<= monitor-promise-base
promise = check_command_execute
name = software-release-deployment-promise.py
config-command =${software-release-deployment-bin:output}
[resilient-software-release-information]
recipe = slapos.recipe.template
url = {{ software_release_information_template }}
output = ${directory:cgi-bin}/resilient_software_release_information.py
resilient-log-url = ${monitor-publish-parameters:monitor-base-url}/log/${importer:resilient-log-basename}
[slap-parameter]
auto-deploy-instance = false
auto-deploy = true
[resilient-publish-connection-parameter]
monitor-base-url = ${monitor-publish-parameters:monitor-base-url}
monitor-url = ${monitor-publish-parameters:monitor-url}
monitor-user = ${monitor-publish-parameters:monitor-user}
monitor-password = ${monitor-publish-parameters:monitor-password}
[monitor-instance-parameter]
monitor-httpd-port = 8360
monitor-title = {{ slapparameter_dict.get('name', 'Webrunner import') }}
cors-domains = {{ slapparameter_dict.get('monitor-cors-domains', 'monitor.app.officejs.com') }}
{% if slapparameter_dict.get('monitor-username', '') -%}
username = {{ slapparameter_dict['monitor-username'] }}
{% endif -%}
{% if slapparameter_dict.get('monitor-password', '') -%}
password = {{ slapparameter_dict['monitor-password'] }}
{% endif -%}
instance-configuration =
raw takeover-url http://[${resilient-web-takeover-httpd-configuration-file:listening-ip}]:${resilient-web-takeover-httpd-configuration-file:listening-port}/
raw takeover-password ${resilient-web-takeover-password:passwd}
configuration-file-path = ${buildout:directory}/knowledge0.cfg
[monitor-conf-parameters]
private-path-list +=
$${logrotate-directory:logrotate-backup}
[post-notification-run]
recipe = slapos.cookbook:wrapper
command-line = {{ software_release_bin }}/runner-importer-post-notification-run --diff-file ${:diff-file} --proof-signature-file ${:proof-signature-file} --srv-path ${directory:srv} --backup-path ${directory:backup}
wrapper-path = ${rootdirectory:bin}/post-notification-run
output = ${:wrapper-path}
mode = 0700
{
"$schema": "http://json-schema.org/draft-04/schema#",
"type": "object",
"properties": {
"instance-name": {
"title": "Instance Name",
"description": "Name of the instance, to show in the window title",
"type": "string"
},
"custom-frontend-backend-url": {
"title": "Custom Frontend Backend URL",
"description": "return an ipv4 frontend of the given ipv6(+optional port)",
"type": "string",
"format": "uri"
},
"custom-frontend-backend-type": {
"title": "Custom Frontend Backend Type",
"description": "The type of the frontend slave instance to ask",
"type": "string",
"enum": [
"zope"
]
},
"user-authorized-key": {
"title": "User Authorized Key",
"description": "SSH public key in order to connect to the SSH server of this runner instance.",
"textarea": true,
"type": "string"
},
"instance-amount": {
"title": "Partition Amount",
"description": "Number of slappart to deploy inside the runner (default is 10). Needs instance to be restarted.",
"type": "integer",
"minimum": 1,
"maximum": 40
},
"auto-deploy": {
"title": "Automatically Deploy Software",
"description": "Authorizes the software declared with 'slapos-software' to be automatically deployed, or not. Needs instance to be restarted. (default is false)",
"type": "boolean"
},
"auto-deploy-instance": {
"title": "Automatically Deploy Instances",
"description": "Prevent the runner from deploying and starting instances. Needs instance to be restarted. It is set to false for instances of type 'import' in resiliency in any case (default is false)",
"type": "boolean"
},
"autorun": {
"title": "Automatically Run Sofware/Instance",
"description": "Let automaticaly build and run a declared software with 'slapos-software'. Only works if 'slapos-software' is set, and 'auto-deploy' is true. Needs instance to be restarted. (default is false)",
"type": "boolean"
},
"slapos-software": {
"title": "Pre-selected Software Release",
"description": "a relative path from the slapos git repo to a folder containing a software release, which will be automaticaly deployed while the runner instanciation, and only if the parameter auto-deploy is set to 'true'. For example: 'software/helloworld",
"type": "string"
},
"slapos-repository": {
"title": "SlapOS Git Repository URL",
"description": "url of the default git repository that will be download by the runner while its instanciation. Will be cloned in a directory named 'slapos' (default is https://lab.nexedi.com/nexedi/slapos.git)",
"type": "string",
"format": "uri",
"pattern": "^(http|https|ftp)://"
},
"slapos-reference": {
"title": "SlapOS Git Branch Name",
"description": "Branch or hash on which the default repository will checkout (default is master)",
"type": "string"
},
"slapos-software-type": {
"title": "Deployed Instance Software Type",
"description": "Software type of your instance inside the runner",
"type": "string"
},
"cpu-usage-ratio": {
"title": "CPU Usage Ratio",
"description": "Ratio of the CPU use for compilation, if value is set to n, compilation will use number-of-cpu/n of cpus (need instance restart)",
"type": "integer",
"default": 4
},
"no-ipv4-frontend": {
"title": "No IPv4 frontend",
"description": "Prevent the slaprunner to order an IPv4 frontend for itself",
"enum": [
"true",
"false"
],
"default": "false"
},
"custom-frontend-basic-auth": {
"title": "Custom Frontend Basic Auth",
"description": "if the ip given with 'custom-frontend-backend-url' is secure, set it to true for the promise do not fail",
"type": "boolean"
},
"custom-frontend-instance-guid": {
"title": "Custom Frontend Instance GUID",
"description": "Instance guid of the frontend you whish to use",
"type": "string"
},
"custom-frontend-software-type": {
"title": "Custom Frontend Software Type",
"description": "SoftwareType of the frontend you request (default is RootSoftwareInstance)",
"type": "string",
"enum": [
"RootSoftwareInstance",
"default"
]
},
"custom-frontend-software-url": {
"title": "Custom Frontend Software URL",
"description": "Software Url of the frontend you request (ie.: http://git.erp5.org/gitweb/slapos.git/blob_plain/HEAD:/software/apache-frontend/software.cfg)",
"type": "string",
"format": "uri"
},
"check-custom-frontend-promise": {
"title": "Check Custom Frontend Promise",
"description": "Enable a promise to check that HTTP frontend created from custom-frontend-backend-url is available",
"type": "string",
"enum": [
"true",
"false"
],
"default": "false"
},
"slaprunner-httpd-port": {
"title": "Webrunner Server Port",
"description": "Port of the Apache server serving the Webrunner interface.",
"type": "integer",
"default": 9686
},
"monitor-httpd-port": {
"title": "Monitor Port",
"description": "Port of the Apache server serving the monitoring interface.",
"type": "integer",
"default": 8386
},
"monitor-interface-url": {
"title": "Monitor Web Interface URL",
"description": "Give Url of HTML web interface that will be used to render this monitor instance.",
"type": "string",
"format": "uri",
"default": "https://monitor.app.officejs.com"
},
"monitor-cors-domains": {
"title": "Monitor CORS domains",
"description": "List of cors domains separated with space. Needed for ajax query on this monitor instance from a different domain.",
"type": "string",
"default": "monitor.app.officejs.com"
}
}
}
{
"$schema": "http://json-schema.org/draft-04/schema#",
"description": "Values returned by Runner instanciation",
"properties": {
"backend_url": {
"description": "ipv6 url to directly access your runner",
"type": "string"
},
"access_url": {
"description": "Url to set up your account and then login into your webrunner",
"type": "string"
},
"url": {
"description": "Url to access your runner once you set your account",
"type": "string"
},
"ssh_command": {
"description": "SSH command used to access your runner in ssh when you provided a ssh public key",
"type": "string"
},
"monitor_url": {
"description": "Url to access the monitoring interface of your instance. Use this url to get the recovery code needed to set up your runner",
"type": "string"
},
"webdav_url": {
"description": "Url for webdav access of your runner it access the root of your runner. Same users as the one for the runner",
"type": "string"
},
"public_url": {
"description": "Url of the public folder of your runner",
"type": "string"
}
},
"type": "object"
}
{
"type": "object",
"$schema": "http://json-schema.org/draft-04/schema",
"allOf": [
{
"$ref": "instance-runner-input-schema.json#/"
},
{
"properties": {
"resilient-clone-number": {
"title": "Amount of backup(s) to create",
"description": "Amount of backup(s) to create. Each backup consists of a Pull Backup Server and a clone.",
"type": "integer",
"default": 1,
"minimum": 0,
"maximum": 2,
"optional": true
},
"-sla-runner0-computer_guid": {
"title": "Target computer for main instance",
"description": "Target computer GUID for main instance.",
"type": "string",
"optional": true
},
"-sla-runner1-computer_guid": {
"title": "Target computer for first clone",
"description": "Target computer for first clone instance.",
"type": "string",
"optional": true
},
"-sla-pbs1-computer_guid": {
"title": "Target computer for first PBS",
"description": "Target computer for first PBS instance.",
"type": "string",
"optional": true
},
"-sla-runner2-computer_guid": {
"title": "Target computer for second clone",
"description": "Target computer for second clone instance.",
"type": "string",
"optional": true
},
"-sla-pbs2-computer_guid": {
"title": "Target computer for second PBS",
"description": "Target computer for second PBS instance.",
"type": "string",
"optional": true
},
"resiliency-backup-periodicity": {
"title": "Periodicity of backup",
"description": "Periodicity of backup, in cron format.",
"type": "string",
"optional": true
},
"remove-backup-older-than": {
"title": "Remove backups older than...",
"description": "Remove all the backups in PBS that are older than specified value. It should be rdiff-backup-compatible.",
"type": "string",
"default": "2W",
"optional": true
}
}
}
]
}
[buildout]
common-runner-parts =
nginx_conf
nginx-launcher
certificate-authority
ca-nginx
certificate-authority-service
ca-nginx-service
logrotate-entry-nginx
gunicorn-launcher
gunicorn-graceful
publish-connection-information
slaprunner-promise
logrotate-entry-apache-httpd
apache-httpd-promise
slaprunner-supervisord-wrapper
runner-sshd-add-authorized-key
runner-sshd-promise
runner-sshd-service
runtestsuite
shellinabox
shellinabox-service
template-slapuser-script
slapos-cfg
slapformat-definition.cfg
cron-entry-prepare-software
deploy-instance-parameters
instance-software
instance-software-type
minishell-cwd
bash-profile
supervisord-wrapper
supervisord-promise
slapos-supervisord-promise
slapos-proxy-promise
logrotate-entry-supervisord
logrotate-entry-slapgrid
httpd-graceful-wrapper
{% if slapparameter_dict.get('no-ipv4-frontend', 'false') == 'false' %}
slaprunner-frontend-promise
httpd-frontend-promise
{% endif %}
{% if slapparameter_dict.get('custom-frontend-backend-url') %}
custom-frontend-url-ready-promise
{% if slapparameter_dict.get('check-custom-frontend-promise', 'false') == 'true' %}
custom-frontend-promise
{% endif %}
{% endif %}
project-link
## Monitoring part
monitor-base
monitor-check-webrunner-internal-instance
parts = $${:common-runner-parts}
extends =
${monitor2-template:output}
${template-logrotate-base:output}
[slap-network-information]
local-ipv4 = $${slap-configuration:ipv4-random}
global-ipv6 = $${slap-configuration:ipv6-random}
{% if slapparameter_dict.get('custom-frontend-backend-url') -%}
[request-custom-frontend]
recipe = slapos.cookbook:requestoptional
software-url = {{ slapparameter_dict.get('custom-frontend-software-url', 'http://git.erp5.org/gitweb/slapos.git/blob_plain/HEAD:/software/apache-frontend/software.cfg') }}
software-type = {{ slapparameter_dict.get('custom-frontend-software-type', 'RootSoftwareInstance') }}
shared = true
name = Custom Web Frontend
server-url = $${slap-connection:server-url}
key-file = $${slap-connection:key-file}
cert-file = $${slap-connection:cert-file}
computer-id = $${slap-connection:computer-id}
partition-id = $${slap-connection:partition-id}
{% if slapparameter_dict.get('custom-frontend-instance-guid') -%}
sla-instance_guid = $${slap-parameter:frontend-instance-guid}
{% endif -%}
{% set custom_frontend_backend_type = slapparameter_dict.get('custom-frontend-backend-type') -%}
{% if custom_frontend_backend_type -%}
config-type = {{ custom_frontend_backend_type }}
{% endif -%}
config-url = {{ slapparameter_dict.get('custom-frontend-backend-url') }}
return = site_url domain
[custom-frontend-promise]
<= monitor-promise-base
promise = check_url_available
name = custom_frontend_promise.py
config-url = https://$${request-custom-frontend:connection-domain}
{% if slapparameter_dict.get('custom-frontend-basic-auth') -%}
config-http-code = 401
{% endif -%}
[custom-frontend-url-ready-promise-bin]
recipe = slapos.recipe.template
frontend-url = https://$${request-custom-frontend:connection-domain}
output = $${directory:bin}/custom_frontend_ready_promise
inline =
#!{{ dash_executable_location }}
URL="$${:frontend-url}"
CODE=$({{ curl_executable_location }} -g -k -sL $URL -w %{http_code} --max-time 5 -o /dev/null)
if [ $? -eq 3 ]; then
echo "Custom frontend URL malformed: $URL." >&2
exit 1
fi
[custom-frontend-url-ready-promise]
<= monitor-promise-base
promise = check_command_execute
name = custom_frontend_ready_promise.py
config-command = $${custom-frontend-url-ready-promise-bin:output}
[publish-connection-information]
custom-frontend-url = $${custom-frontend-url-ready-promise-bin:frontend-url}
{% endif %}
# Create all needed directories
[directory]
recipe = slapos.cookbook:mkdirectory
home = $${buildout:directory}
etc = $${:home}/etc/
var = $${:home}/var/
# This srv path has an extra slash, which will cause runnerdirectory:home
# to be .../srv//runner/.. but for compatibility reasons we don't fix this,
# because this is the path that will be used as software URL installed by
# webrunner and would cause software release hash to become different.
srv = $${:home}/srv/
bin = $${:home}/bin/
tmp = $${:home}/tmp/
sshkeys = $${:srv}/sshkeys
services = $${:etc}/service/
scripts = $${:etc}/run/
ssh = $${:etc}/ssh/
log = $${:var}/log/
run = $${:var}/run/
backup = $${:srv}/backup/
test = $${:etc}/test/
nginx-data = $${:srv}/nginx
ca-dir = $${:srv}/ssl
project = $${:srv}/runner/project
cgi-bin = $${:srv}/cgi-bin
[runnerdirectory]
recipe = slapos.cookbook:mkdirectory
home = $${directory:srv}/runner/
test = $${directory:srv}/test/
project = $${:home}/project
public = $${:home}/public
software-root = {{ slapparameter_dict.get('software-root', '$${:home}/software') }}
instance-root = $${:home}/instance
shared-root = $${slap-parameter:buildout-shared-folder}
project-test = $${:test}/project
software-test = $${:test}/software
instance-test = $${:test}/instance
sessions = $${buildout:directory}/.sessions
[project-link]
recipe = slapos.cookbook:symbolic.link
target-directory = $${directory:srv}
link-binary = $${runnerdirectory:project}
[slaprunner]
slaprunner = ${buildout:bin-directory}/slaprunner
slapos = ${buildout:bin-directory}/slapos
slapproxy = ${buildout:bin-directory}/slapproxy
supervisor = ${buildout:bin-directory}/slapgrid-supervisorctl
git-binary = ${git:location}/bin/git
root_check = false
slapos.cfg = $${directory:etc}/slapos.cfg
slapformat-definition.cfg = $${directory:etc}/slapformat-definition.cfg
working-directory = $${runnerdirectory:home}
project-directory = $${runnerdirectory:project}
instance_root = $${runnerdirectory:instance-root}
software_root = $${runnerdirectory:software-root}
shared_root = $${runnerdirectory:shared-root}
buildout-shared-part-list-dump = ${template-buildout-shared-part-list:output}
pidfile-software = $${directory:run}/slapgrid-cp.pid
pidfile-instance = $${directory:run}/slapgrid-sr.pid
public_key = $${runner-sshd-ssh-host-rsa-key:output}
instance-monitor-url = https://[$${:ipv6}]:$${slap-parameter:monitor-httpd-port}
etc_dir = $${directory:etc}
log_dir = $${directory:log}
run_dir = $${directory:run}
ipv4 = $${slap-network-information:local-ipv4}
ipv6 = $${slap-network-information:global-ipv6}
instance_root = $${runnerdirectory:instance-root}
proxy_port = 50000
runner_port = 50005
partition-amount = $${slap-parameter:instance-amount}
wrapper = $${directory:services}/slaprunner
debug = $${slap-parameter:debug}
access-url = https://[$${:ipv6}]:$${:runner_port}
supervisord_config = $${directory:etc}/supervisord.conf
supervisord_server = http://$${supervisord:server}
proxy_database = $${slaprunner:working-directory}/proxy.db
console = False
verbose = False
debug = False
auto_deploy = $${slap-parameter:auto-deploy}
auto_deploy_instance = $${slap-parameter:auto-deploy-instance}
autorun = $${slap-parameter:autorun}
knowledge0_file = $${monitor-instance-parameter:configuration-file-path}
minishell_cwd_file = $${directory:etc}/.minishell-cwd
minishell_history_file = $${directory:etc}/.minishell_history
software_info_json = $${runnerdirectory:home}/software_info.json
instance_info_json = $${runnerdirectory:home}/instance_info.json
path = $${shell-environment:path}
instance_name = $${slap-parameter:instance-name}
default_repository = $${slap-parameter:slapos-repository}
default_repository_branch = $${slap-parameter:slapos-reference}
#---------------------------
#--
#-- supervisord managing slaprunner instance processes
[slaprunner-supervisord-wrapper]
recipe = slapos.cookbook:wrapper
command-line = ${buildout:bin-directory}/slapos node supervisord --cfg=$${slaprunner:slapos.cfg} --nodaemon
wrapper-path = $${directory:services}/slaprunner-supervisord
hash-existing-files = $${buildout:directory}/software_release/buildout.cfg
[test-runner]
<= slaprunner
slapos.cfg = $${directory:etc}/slapos-test.cfg
working-directory = $${runnerdirectory:test}
project-directory = $${runnerdirectory:project-test}
software_root = $${runnerdirectory:software-test}
instance_root = $${runnerdirectory:instance-test}
proxy_port = 8602
etc_dir = $${directory:test}
autorun = False
auto_deploy = True
[runtestsuite]
recipe = slapos.cookbook:wrapper
arguments = --server_url=$${slap-connection:server-url} --key_file=$${slap-connection:key-file} --cert_file=$${slap-connection:cert-file} --computer_id=$${slap-connection:computer-id} --partition_id=$${slap-connection:partition-id}
command-line = ${buildout:bin-directory}/slaprunnertest $${:arguments}
wrapper-path = $${directory:bin}/runTestSuite
environment = PATH=$${shell-environment:path}
RUNNER_CONFIG=$${slapos-cfg:output}
# Deploy openssh-server
[runner-sshd-port]
recipe = slapos.cookbook:free_port
minimum = 22222
maximum = 22231
ip = $${slap-network-information:global-ipv6}
[runner-sshd-config]
recipe = slapos.recipe.template:jinja2
output = $${directory:etc}/runner-sshd.conf
path_pid = $${directory:run}/runner-sshd.pid
inline =
PidFile $${:path_pid}
Port $${runner-sshd-port:port}
ListenAddress $${slap-network-information:global-ipv6}
Protocol 2
HostKey $${runner-sshd-ssh-host-rsa-key:output}
HostKey $${runner-sshd-ssh-host-ecdsa-key:output}
PasswordAuthentication no
PubkeyAuthentication yes
AuthorizedKeysFile $${buildout:directory}/.ssh/authorized_keys
ForceCommand cd $${directory:home}; if [ -z "$SSH_ORIGINAL_COMMAND" ]; then HOME=$${directory:home} $${shell-environment:shell} -l; else HOME=$${directory:home} SHELL=$${shell-environment:shell} PATH=$${shell-environment:path} eval "$SSH_ORIGINAL_COMMAND"; fi
Subsystem sftp ${openssh:location}/libexec/sftp-server
[runner-sshd-service]
recipe = slapos.cookbook:wrapper
command-line = ${openssh:location}/sbin/sshd -D -e -f $${runner-sshd-config:output}
hash-existing-files = $${buildout:directory}/software_release/buildout.cfg
wrapper-path = $${directory:services}/runner-sshd
[runner-sshd-add-authorized-key]
recipe = slapos.cookbook:dropbear.add_authorized_key
home = $${buildout:directory}
key = $${slap-parameter:user-authorized-key}
[runner-sshd-ssh-keygen-base]
recipe = plone.recipe.command
output = $${directory:etc}/$${:_buildout_section_name_}
command = ${openssh-output:keygen} -f $${:output} -N '' $${:extra-args}
[runner-sshd-ssh-host-rsa-key]
<=runner-sshd-ssh-keygen-base
extra-args=-t rsa
[runner-sshd-ssh-host-ecdsa-key]
<=runner-sshd-ssh-keygen-base
extra-args=-t ecdsa -b 521
[runner-sshd-publickey-fingerprint]
# fingerprint for ssh url, see
# https://tools.ietf.org/id/draft-salowey-secsh-uri-00.html#connparam
# https://winscp.net/eng/docs/session_url#hostkey
recipe = slapos.recipe.build
init =
import os
import subprocess
from six.moves.urllib.parse import quote
keyfile = self.buildout['runner-sshd-ssh-host-ecdsa-key']['output']
if os.path.isfile(keyfile):
x = subprocess.check_output(('${openssh-output:keygen}', '-lf', keyfile))
x = x.split()[1]
assert x.startswith(b'SHA256:'), x
# format is host-key-alg-fingerprint, but we know that
# $${runner-sshd-ssh-host-ecdsa-key:output} is rsa so for host-key-alg
# we just use use rsa.
options['fingerprint'] = "ssh-rsa-" + quote(x[7:], safe='')
else:
# This command will always fail on first buildout run, because it is
# evaluated at buildout recipes __init__ step, but the key file is created
# later at install step.
options['fingerprint'] = "NotReady"
#---------------------------
#--
#-- Set nginx frontend
[tempdirectory]
recipe = slapos.cookbook:mkdirectory
client_body_temp_path = $${directory:tmp}/client_body_temp_path
proxy_temp_path = $${directory:tmp}/proxy_temp_path
fastcgi_temp_path = $${directory:tmp}/fastcgi_temp_path
uwsgi_temp_path = $${directory:tmp}/uwsgi_temp_path
scgi_temp_path = $${directory:tmp}/scgi_temp_path
[nginx-frontend]
# Options
nb_workers = 5
# Network
local-ip = $${slap-network-information:local-ipv4}
global-ip = $${slap-network-information:global-ipv6}
global-port = $${slaprunner:runner_port}
# Backend
runner-ip = $${slaprunner:ipv4}
runner-port = $${slaprunner:runner_port}
# SSL
ssl-certificate = $${ca-nginx:cert-file}
ssl-key = $${ca-nginx:key-file}
# Log
path_pid = $${directory:run}/nginx.pid
path_log = $${directory:log}/nginx.log
path_access_log = $${directory:log}/nginx.access.log
path_error_log = $${directory:log}/nginx.error.log
path_tmp = $${directory:tmp}/
nginx_prefix = $${buildout:directory}
# Config files
path_nginx_conf = $${directory:etc}/nginx.conf
# Executables
bin_nginx = ${nginx-webdav:location}/sbin/nginx
bin_launcher = $${directory:bin}/launcher
# Utils
path_shell = ${dash:location}/bin/dash
# Misc.
etc_dir = $${directory:etc}
work_dir = $${slaprunner:working-directory}
[nginx_conf]
recipe = slapos.recipe.template:jinja2
url = ${template_nginx_conf:target}
output = $${nginx-frontend:path_nginx_conf}
context =
key shellinabox_socket shellinabox:socket
key socket gunicorn:socket
section param_nginx_frontend nginx-frontend
section param_tempdir tempdirectory
[nginx-launcher]
recipe = slapos.recipe.template:jinja2
url = ${template_launcher:target}
output = $${nginx-frontend:bin_launcher}
context =
section param_nginx_frontend nginx-frontend
[logrotate-entry-nginx]
<= logrotate-entry-base
name = nginx
log = $${directory:log}/nginx.access.log $${directory:log}/nginx.error.log
post = kill -USR1 $(cat $${buildout:directory}/var/run/nginx.pid)
[httpd-parameters]
path_pid = $${directory:run}/httpd.pid
path_error_log = $${directory:log}/httpd-error.log
path_access_log = $${directory:log}/httpd-access.log
# XXX Use ca-nginx, no need to regenerate certificate
cert_file = $${ca-nginx:cert-file}
key_file = $${ca-nginx:key-file}
global_ip = $${slap-network-information:global-ipv6}
global_port = $${slap-parameter:slaprunner-httpd-port}
working_directory = $${slaprunner:working-directory}
dav_lock = $${directory:var}/WebDavLock
htpasswd_file = $${directory:etc}/.htpasswd
etc_dir = $${directory:etc}
var_dir = $${directory:var}
project_folder = $${directory:project}
runner_home = $${runnerdirectory:home}
httpd_cors_file = $${slaprunner-httpd-cors:location}
[httpd-conf]
recipe = slapos.recipe.template:jinja2
url = ${template_httpd_conf:target}
output = $${directory:etc}/httpd.conf
context =
section parameters httpd-parameters
[apache-httpd]
recipe = slapos.cookbook:wrapper
apache-executable = ${apache:location}/bin/httpd
wrapper-path = $${directory:services}/slaprunner-httpd
command-line = $${:apache-executable} -f $${httpd-conf:output} -DFOREGROUND
access-url = https://[$${httpd-parameters:global_ip}]:$${httpd-parameters:global_port}
wait-for-files =
$${ca-nginx:cert-file}
$${ca-nginx:key-file}
hash-existing-files = $${buildout:directory}/software_release/buildout.cfg
[logrotate-entry-apache-httpd]
<= logrotate-entry-base
name = apache
log = $${directory:log}/httpd-access.log $${directory:log}/httpd-error.log
post = test ! -s $${buildout:directory}/var/run/httpd.pid || ${buildout:bin-directory}/slapos-kill --pidfile $${buildout:directory}/var/run/httpd.pid -s USR1
[httpd-graceful-wrapper]
recipe = collective.recipe.template
input = inline:
#!/bin/sh
exec kill -USR1 $(cat $${httpd-parameters:path_pid})
output = $${directory:scripts}/slaprunner-httpd-graceful
mode = 700
[apache-httpd-promise]
<= monitor-promise-base
promise = check_url_available
name = $${:filename}.py
filename = apache-httpd-listening-on-tcp
config-url = $${apache-httpd:access-url}
[slaprunner-httpd-cors]
recipe = plone.recipe.command
command = if [ ! -f $${:location} ]; then touch $${:location}; fi
location = $${directory:etc}/$${:filename}
filename = slaprunner-httpd-cors.cfg
stop-on-error = true
#--------------------
#--
#-- WSGI
[gunicorn]
bin_gunicorn = ${buildout:bin-directory}/gunicorn
bin_launcher = $${directory:services}/gunicorn
path_shell = ${dash:location}/bin/dash
socket = $${directory:tmp}/flaskserver.sock
path_pid = $${directory:run}/gunicorn.pid
[gunicorn-launcher]
recipe = slapos.cookbook:wrapper
command-line = $${gunicorn:bin_gunicorn} slapos.runner.run:app -p $${gunicorn:path_pid} -b unix:$${gunicorn:socket} -e RUNNER_CONFIG=$${slaprunner:slapos.cfg} --error-logfile $${directory:log}/$${:error-log-file} --timeout 200 --threads 3 --log-level error --preload
error-log-file = gunicorn-error.log
wrapper-path = $${gunicorn:bin_launcher}
environment = PATH=$${shell-environment:path}
RUNNER_CONFIG=$${slaprunner:slapos.cfg}
LANG=en_GB.UTF-8
hash-existing-files = $${buildout:directory}/software_release/buildout.cfg
[gunicorn-graceful]
recipe = slapos.cookbook:wrapper
command-line = ${buildout:bin-directory}/killpidfromfile $${gunicorn:path_pid} SIGHUP
wrapper-path = $${directory:scripts}/gunicorn-graceful
#--------------------
#--
#-- ssl certificates
[certificate-authority]
recipe = slapos.cookbook:certificate_authority
openssl-binary = ${openssl:location}/bin/openssl
ca-dir = $${directory:ca-dir}
requests-directory = $${cadirectory:requests}
wrapper = $${directory:bin}/certificate_authority
ca-private = $${cadirectory:private}
ca-certs = $${cadirectory:certs}
ca-newcerts = $${cadirectory:newcerts}
ca-crl = $${cadirectory:crl}
[cadirectory]
recipe = slapos.cookbook:mkdirectory
requests = $${directory:ca-dir}/requests/
private = $${directory:ca-dir}/private/
certs = $${directory:ca-dir}/certs/
newcerts = $${directory:ca-dir}/newcerts/
crl = $${directory:ca-dir}/crl/
[ca-nginx]
<= certificate-authority
recipe = slapos.cookbook:certificate_authority.request
key-file = $${cadirectory:certs}/nginx_frontend.key
cert-file = $${cadirectory:certs}/nginx_frontend.crt
executable = $${nginx-launcher:output}
wrapper = $${directory:bin}/nginx-frontend
# Put domain name
name = example.com
[ca-nginx-service]
recipe = slapos.cookbook:wrapper
command-line = $${directory:bin}/nginx-frontend
wrapper-path = $${directory:services}/nginx-frontend
hash-existing-files = $${buildout:directory}/software_release/buildout.cfg
[certificate-authority-service]
recipe = slapos.cookbook:wrapper
command-line = $${directory:bin}/certificate_authority
wrapper-path = $${directory:services}/certificate_authority
hash-existing-files = $${buildout:directory}/software_release/buildout.cfg
#--------------------
#--
#-- Request frontend
{% if slapparameter_dict.get('no-ipv4-frontend', 'false') == 'false' -%}
[request-frontend]
<= slap-connection
recipe = slapos.cookbook:requestoptional
name = SlapRunner Frontend
# XXX We have hardcoded SR URL here.
software-url = http://git.erp5.org/gitweb/slapos.git/blob_plain/HEAD:/software/apache-frontend/software.cfg
shared = true
config-url = $${slaprunner:access-url}
config-domain = $${slap-parameter:frontend-domain}
return = site_url domain
[slaprunner-frontend-promise]
<= monitor-promise-base
promise = check_url_available
name = slaprunner_frontend.py
config-url = https://$${request-frontend:connection-domain}/login
[request-httpd-frontend]
<= slap-connection
recipe = slapos.cookbook:requestoptional
# XXX - Unfortunately, we still call webrunner httpd frontend "Monitor Frontend" otherwise
# buildout will ignore previous frontend that was created and create a new one (in case of upgrade)
name = Monitor Frontend
# XXX We have hardcoded SR URL here.
software-url = http://git.erp5.org/gitweb/slapos.git/blob_plain/HEAD:/software/apache-frontend/software.cfg
shared = true
config-url = $${apache-httpd:access-url}
config-domain =
return = secure_access domain
[httpd-frontend-promise]
<= monitor-promise-base
promise = check_url_available
name = slaprunner-apache-http-frontend.py
config-url = $${request-httpd-frontend:connection-secure_access}
{% endif %}
[htpasswd]
recipe = slapos.cookbook:generate.password
storage-path = $${directory:etc}/.pwd
bytes = 8
[runner-htpasswd]
recipe = plone.recipe.command
stop-on-error = true
htpasswd-path = $${monitor-directory:etc}/.htpasswd
command = if [ ! -f "$${:htpasswd-path}" ]; then ${apache:location}//bin/htpasswd -cb $${:htpasswd-path} $${:user} $${:password}; fi
update-command = $${:command}
user = admin
{% if slapparameter_dict.get('monitor-password', '') -%}
password = {{ slapparameter_dict['monitor-password'] }}
{% else -%}
password = $${htpasswd:passwd}
{% endif -%}
#--------------------------------------
#--
#-- Send information to SlapOS Master
[user-info]
recipe = slapos.cookbook:userinfo
[publish-connection-information]
recipe = slapos.cookbook:publish
backend-url = $${slaprunner:access-url}
init-user = $${runner-htpasswd:user}
init-password = $${runner-htpasswd:password}
ssh-command = ssh $${user-info:pw-name}@$${slap-network-information:global-ipv6} -p $${runner-sshd-port:port}
ssh-url = ssh://$${user-info:pw-name};fingerprint=$${runner-sshd-publickey-fingerprint:fingerprint}@[$${slap-network-information:global-ipv6}]:$${runner-sshd-port:port}
monitor-base-url = $${monitor-publish-parameters:monitor-base-url}
{% if slapparameter_dict.get('no-ipv4-frontend', 'false') == 'false' -%}
url = https://$${request-frontend:connection-domain}
webdav-url = $${request-httpd-frontend:connection-secure_access}/share/
public-url = $${request-httpd-frontend:connection-secure_access}/public/
{% else %}
url = $${slaprunner:access-url}
webdav-url = $${apache-httpd:access-url}/share/
public-url = $${apache-httpd:access-url}/public/
{% endif %}
{% if slapparameter_dict.get('instance-type', '') != 'resilient' -%}
{% set monitor_interface_url = slapparameter_dict.get('monitor-interface-url', 'https://monitor.app.officejs.com') -%}
monitor-setup-url = {{ monitor_interface_url }}/#page=settings_configurator&url=$${monitor-publish-parameters:monitor-url}&username=$${monitor-publish-parameters:monitor-user}&password=$${monitor-publish-parameters:monitor-password}
{% else -%}
monitor-url = $${monitor-publish-parameters:monitor-url}
monitor-user = $${monitor-publish-parameters:monitor-user}
monitor-password = $${monitor-publish-parameters:monitor-password}
{% endif -%}
#---------------------------
#--
#-- Deploy promises scripts
[slaprunner-promise]
<= monitor-promise-base
promise = check_socket_listening
name = slaprunner.py
config-host = $${slaprunner:ipv6}
config-port = $${slaprunner:runner_port}
[runner-sshd-promise]
<= monitor-promise-base
promise = check_socket_listening
name = runner-sshd.py
config-host = $${slap-network-information:global-ipv6}
config-port = $${runner-sshd-port:port}
[slap-parameter]
# Default value if no ssh key is specified
user-authorized-key =
# Default value of instances number in slaprunner
instance-amount = 10
debug = false
frontend-domain =
slapos-repository = https://lab.nexedi.com/nexedi/slapos.git
slapos-software =
slapos-software-type =
slapos-reference = 1.0
auto-deploy = false
auto-deploy-instance = true
autorun = false
slaprunner-httpd-port = 9686
instance-name =
monitor-cors-domains =
monitor-interface-url =
monitor-httpd-port = 8386
buildout-shared-folder = $${runnerdirectory:home}/shared
{% for k, v in slapparameter_dict.items() -%}
{% if k == 'user-authorized-key' and v -%}
{% set key_list = v.split('\n') -%}
{{ k }} =
{{ key_list | join('\n ') }}
{% else -%}
{{ k }} = {{ v }}
{% endif -%}
{% endfor -%}
[slapos-cfg]
recipe = slapos.recipe.template:jinja2
url = ${template-slapos-cfg:target}
output = $${slaprunner:slapos.cfg}
context =
section slaprunner slaprunner
import codecs codecs
raw buildout_shared_part_list_dump $${slaprunner:buildout-shared-part-list-dump}
[slapos-test-cfg]
recipe = slapos.recipe.template:jinja2
url = ${template-slapos-cfg:target}
output = $${test-runner:slapos.cfg}
context =
section slaprunner test-runner
[slapformat-definition.cfg]
recipe = slapos.recipe.template:jinja2
url = ${template-slapformat-definition.cfg:target}
output = $${slaprunner:slapformat-definition.cfg}
context =
section slaprunner slaprunner
key partition_user user-info:pw-name
import codecs codecs
raw buildout_shared_part_list_dump $${slaprunner:buildout-shared-part-list-dump}
[shellinabox]
recipe = slapos.recipe.template
# We cannot use slapos.cookbook:wrapper here because this recipe escapes too much
socket = $${directory:run}/siab.sock
output = $${directory:bin}/shellinaboxd
inline =
#!/bin/sh
exec ${shellinabox:location}/bin/shellinaboxd \
--unixdomain-only=$${:socket}:$(id -u):$(id -g):0600 \
--service "/:$(id -u):$(id -g):HOME:$${shell-environment:shell} -l"
[shellinabox-service]
recipe = slapos.cookbook:wrapper
command-line = $${shellinabox:output}
wrapper-path = $${directory:services}/shellinaboxd
hash-existing-files = $${buildout:directory}/software_release/buildout.cfg
[shell-environment]
shell = ${bash:location}/bin/bash
path = ${nano:location}/bin:${vim:location}/bin:${screen:location}/bin:${tmux:location}/bin:${git:location}/bin:${curl:location}/bin:${python:location}/bin:${tig:location}/bin:${zip:location}/bin:${mosh:location}/bin:${bash:location}/bin:${fish-shell:location}/bin:$${directory:bin}:/usr/bin:/bin/
[prepare-software]
recipe = slapos.cookbook:wrapper
command-line = ${curl:location}/bin/curl -g https://[$${slaprunner:ipv6}]:$${slaprunner:runner_port}/isSRReady --max-time 1 --insecure
wrapper-path = $${directory:scripts}/prepareSoftware
[cron-entry-prepare-software]
<= cron
recipe = slapos.cookbook:cron.d
name = prepare-software
frequency = */2 * * * *
command = $${prepare-software:wrapper-path}
[instance-parameters]
recipe = slapos.recipe.template:jinja2
extensions = jinja2.ext.do
url = ${template-parameters:target}
output = $${directory:etc}/.parameter.xml.default
context =
key slapparameter_dict slap-configuration:configuration
[deploy-instance-parameters]
recipe = plone.recipe.command
stop-on-error = true
parameter-xml = $${directory:etc}/.parameter.xml
command = if [ ! -f $${:parameter-xml} ]; then cp $${instance-parameters:output} $${:parameter-xml}; fi
[instance-software-type]
recipe = plone.recipe.command
stop-on-error = true
# XXX It should not be named with .xml as it is not xml
software-type-path = $${directory:etc}/.software_type.xml
command = if [ ! -f $${:software-type-path} -a "$${slap-parameter:slapos-software-type}" != "" ]; then echo "$${slap-parameter:slapos-software-type}" > $${:software-type-path}; fi
[instance-software]
recipe = plone.recipe.command
stop-on-error = true
command = SR=$${slap-parameter:slapos-software} && if [ -n "$SR" ] && [ ! -f "$${directory:etc}/.project" ]; then echo workspace/slapos/$${slap-parameter:slapos-software}/ > $${directory:etc}/.project; fi
[minishell-cwd]
recipe = plone.recipe.command
command = if [ ! -f $${slaprunner:minishell_cwd_file} ]; then echo $${runnerdirectory:home} > $${slaprunner:minishell_cwd_file}; fi
location = $${slaprunner:minishell_cwd_file}
stop-on-error = true
[bash-profile]
recipe = slapos.recipe.template:jinja2
url = ${template-bash-profile:target}
output = $${buildout:directory}/.bash_profile
context =
raw path $${shell-environment:path}
raw shell $${shell-environment:shell}
key terminfo terminfo:location
key instance_name slap-parameter:instance-name
key workdir runnerdirectory:home
key home directory:home
[terminfo]
location = ${ncurses:location}/share/terminfo/
#---------------------------
#--
#-- supervisord managing slaprunner automation features
[supervisord]
autorestart = false
autostart = false
directory = $${buildout:directory}
exitcodes = 0
logfile = $${directory:log}/supervisord.log
no_logfile = NONE
numprocs = 1
path = $${shell-environment:path}
pidfile = $${directory:run}/supervisord.pid
ip = $${slaprunner:ipv4}
server = $${:ip}:$${:port}
port = 39986
slapgrid-cp = slapgrid-cp
slapgrid-cp-command = $${slaprunner:slapos} node instance --cfg $${:slapos-cfg} --verbose --logfile $${:slapgrid-cp-log}
slapgrid-cp-log = $${runnerdirectory:home}/instance.log
slapgrid-cp-startretries = 0
slapgrid-sr = slapgrid-sr
slapgrid-sr-command = $${slaprunner:slapos} node software --all --cfg $${:slapos-cfg} --verbose --logfile $${:slapgrid-sr-log}
slapgrid-sr-log = $${runnerdirectory:home}/software.log
slapgrid-sr-startretries = 0
slapproxy = slapproxy
slapproxy-autorestart = true
slapproxy-autostart = true
slapproxy-startsecs = 1
slapproxy-command = $${slaprunner:slapos} proxy start --logfile $${:slapproxy-log} --cfg $${:slapos-cfg}
slapproxy-log = $${directory:log}/slapproxy.log
slapformat = slapformat
slapformat-command = $${slaprunner:slapos} node format --cfg $${:slapos-cfg} --verbose --logfile $${:slapformat-log} --now
slapformat-log = $${directory:log}/slapos-node-format.log
slapformat-startretries = 0
slapboot = slapboot
slapboot-command = $${slaprunner:slapos} node boot --cfg $${:slapos-cfg} --verbose --logfile $${:slapboot-log}
slapboot-log = $${directory:log}/slapos-node-boot.log
slapboot-startretries = 0
slapbang = slapbang
slapbang-command = $${slaprunner:slapos} node bang --cfg $${:slapos-cfg} --verbose --logfile $${:slapbang-log}
slapbang-log = $${directory:log}/slapos-node-bang.log
slapbang-startretries = 0
socket_name = unix://$${:socket_path}
socket_path = $${directory:tmp}/supervisord.sock
startsecs = 0
# This file logs errors from listeners. Supervisord has its own logfile.
# Processes should handle their logs by themselves
stderr_logfile = $${directory:log}/supervisord-errors.log
slapos-cfg = $${slaprunner:slapos.cfg}
slapformat-definition.cfg = $${slaprunner:slapformat-definition.cfg}
[supervisord-conf]
recipe = slapos.recipe.template:jinja2
url = ${template-supervisord:target}
output = $${directory:etc}/supervisord.conf
context =
import multiprocessing multiprocessing
section supervisord supervisord
key slapparameter_dict slap-configuration:configuration
key listener_slapgrid listener-slapgrid-bin:output
[listener-slapgrid-bin]
recipe = slapos.recipe.template:jinja2
url = ${template-listener-slapgrid:target}
output = $${directory:bin}/listener_slapgrid.py
context =
section supervisord supervisord
section slaprunner slaprunner
raw python_executable ${buildout:bin-directory}/${python-with-eggs:interpreter}
[supervisord-wrapper]
recipe = slapos.cookbook:wrapper
command-line = ${buildout:bin-directory}/supervisord -c $${supervisord-conf:output} --nodaemon
wrapper-path = $${directory:services}/supervisord
hash-existing-files = $${buildout:directory}/software_release/buildout.cfg
[logrotate-entry-supervisord]
<= logrotate-entry-base
name = supervisord
log = $${directory:log}/slapproxy.log $${directory:log}/supervisord.log $${directory:log}/supervisord-errors.log
post = kill -USR2 $(cat $${buildout:directory}/var/run/supervisord.pid)
[logrotate-entry-slapgrid]
<= logrotate-entry-base
name = slapgrid
log = $${runnerdirectory:home}/instance/*/.slapgrid/log/instance.log $${runnerdirectory:home}/instance/*/.slapgrid/promise/log/*.log
[supervisord-promise]
<= monitor-promise-base
promise = check_socket_listening
name = supervisord.py
config-host = $${slaprunner:ipv4}
config-port = $${supervisord:port}
[slapos-supervisord-promise]
<= monitor-promise-base
promise = check_command_execute
name = instance_supervisord.py
config-command = ${buildout:bin-directory}/slapos node supervisorctl --cfg=$${slaprunner:slapos.cfg} pid
[slapos-proxy-promise]
<= monitor-promise-base
promise = check_socket_listening
name = slaproxy.py
config-host = $${slaprunner:ipv4}
config-port = $${slaprunner:proxy_port}
# XXX Monitor
[monitor-instance-parameter]
monitor-httpd-port = $${slap-parameter:monitor-httpd-port}
{% if slapparameter_dict.get('name', '') -%}
monitor-title = {{ slapparameter_dict['name'] }}
{% endif -%}
cors-domains = {{ slapparameter_dict.get('monitor-cors-domains', 'monitor.app.officejs.com') }}
{% if slapparameter_dict.get('monitor-username', '') -%}
username = {{ slapparameter_dict['monitor-username'] }}
{% endif -%}
password = $${runner-htpasswd:password}
{% if slapparameter_dict.get('monitor-url-list', '') -%}
monitor-url-list = {{ slapparameter_dict['monitor-url-list'] }}
{% endif -%}
instance-configuration =
httpdcors cors-domain $${slaprunner-httpd-cors:location} $${httpd-graceful-wrapper:output}
configuration-file-path = $${buildout:directory}/knowledge0.cfg
[monitor-conf-parameters]
private-path-list +=
$${logrotate-directory:logrotate-backup}
[monitor-check-webrunner-internal-instance]
recipe = slapos.recipe.template
url = ${monitor-check-webrunner-internal-instance:target}
output = $${monitor-directory:bin}/$${:filename}
filename = monitor-check-webrunner-internal-instance
## Slapuser slapos command script
[template-slapuser-script]
recipe = slapos.recipe.template:jinja2
url = ${template-slapuser-script:target}
output = $${buildout:bin-directory}/slapos
context =
raw config_location $${slapos-cfg:output}
raw slapos_python_file_location ${buildout:bin-directory}/slapos
[buildout]
parts =
switch_softwaretype
eggs-directory = ${buildout:eggs-directory}
develop-eggs-directory = ${buildout:develop-eggs-directory}
extends =
${template-resilient-templates:output}
[switch_softwaretype]
recipe = slapos.cookbook:switch-softwaretype
default = $${:runner}
resilient = instance-resilient:output
runner = instance-base-runner:output
runner-import = template-runner-import:output
runner-export = template-runner-export:output
frozen = instance-frozen:output
pull-backup = template-pull-backup:output
# BBB
RootSoftwareInstance = $${:default}
[instance-base-runner]
recipe = slapos.recipe.template:jinja2
url = ${template-runner:output}
output = $${buildout:directory}/template-runner.cfg
extensions = jinja2.ext.do
context = key buildout buildout:bin-directory
key develop_eggs_directory buildout:develop-eggs-directory
key eggs_directory buildout:eggs-directory
key slapparameter_dict slap-configuration:configuration
raw curl_executable_location ${curl:location}/bin/curl
raw dash_executable_location ${dash:location}/bin/dash
raw dcron_executable_location ${dcron:location}/sbin/crond
[instance-resilient]
recipe = slapos.recipe.template:jinja2
url = ${template-resilient:target}
output = $${buildout:directory}/instance-resilient.cfg
extensions = jinja2.ext.do
context = key buildout buildout:bin-directory
key develop_eggs_directory buildout:develop-eggs-directory
key eggs_directory buildout:eggs-directory
key slapparameter_dict slap-configuration:configuration
raw monitor_template ${monitor-template:output}
template-parts-destination = ${template-parts:target}
template-replicated-destination = ${template-replicated:target}
import-list = file parts :template-parts-destination
file replicated :template-replicated-destination
[template-runner-export]
recipe = slapos.recipe.template:jinja2
url = ${instance-runner-export:target}
output = $${buildout:directory}/instance-runner-export.cfg
context =
key pbsready_export_template_path template-pbsready-export:output
key template_runner_path instance-base-runner:output
key slapparameter_dict slap-configuration:configuration
raw software_release_bin ${buildout:bin-directory}
raw backup_wait_time ${exporter-default-configuration:backup_wait_time}
raw monitor_check_resilient_feed_template_path ${template-monitor-check-resilient-feed:target}
raw buildout_executable_location ${buildout:executable}
raw bash_executable_location ${bash:location}/bin/bash
raw rsync_bin_folder ${rsync:location}/bin
[template-runner-import]
recipe = slapos.recipe.template:jinja2
url = ${instance-runner-import:target}
output = $${buildout:directory}/instance-runner-import.cfg
context =
key pbsready_import_template_path template-pbsready-import:output
key template_runner_path instance-base-runner:output
key slapparameter_dict slap-configuration:configuration
raw software_release_bin ${buildout:bin-directory}
raw importer_script_path ${template-runner-import-script:target}
raw buildout_executable_location ${buildout:executable}
raw bash_executable_location ${bash:location}/bin/bash
raw sqlite3_executable_location ${sqlite3:location}/bin/sqlite3
raw rsync_executable_location ${rsync:location}/bin/rsync
raw software_release_information_template ${template-resilient-software-release-information:target}
[slap-configuration]
recipe = slapos.cookbook:slapconfiguration
computer = $${slap-connection:computer-id}
partition = $${slap-connection:partition-id}
url = $${slap-connection:server-url}
key = $${slap-connection:key-file}
cert = $${slap-connection:cert-file}
#! {{ param_nginx_frontend['path_shell'] }}
# BEWARE: This file is operated by slapos node
# BEWARE: It will be overwritten automatically
# Run nginx
exec {{ param_nginx_frontend['bin_nginx'] }} \
-p {{ param_nginx_frontend['nginx_prefix'] }} \
-c {{ param_nginx_frontend['path_nginx_conf'] }}
worker_processes {{ param_nginx_frontend['nb_workers'] }};
pid {{ param_nginx_frontend['path_pid'] }};
error_log {{ param_nginx_frontend['path_error_log'] }};
daemon off;
events {
worker_connections 1024;
accept_mutex off;
}
http {
default_type application/octet-stream;
access_log {{ param_nginx_frontend['path_access_log'] }} combined;
client_max_body_size 10M;
map $http_upgrade $connection_upgrade {
default upgrade;
'' close;
}
server {
listen [{{ param_nginx_frontend['global-ip'] }}]:{{ param_nginx_frontend['global-port'] }} ssl;
server_name _;
ssl_certificate {{ param_nginx_frontend['ssl-certificate'] }};
ssl_certificate_key {{ param_nginx_frontend['ssl-key'] }};
ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
ssl_ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+3DES:HIGH:!aNULL:!MD5;
ssl_prefer_server_ciphers on;
keepalive_timeout 90s;
client_body_temp_path {{ param_tempdir['client_body_temp_path'] }};
proxy_temp_path {{ param_tempdir['proxy_temp_path'] }};
fastcgi_temp_path {{ param_tempdir['fastcgi_temp_path'] }};
uwsgi_temp_path {{ param_tempdir['uwsgi_temp_path'] }};
scgi_temp_path {{ param_tempdir['scgi_temp_path'] }};
location / {
auth_basic "Restricted";
auth_basic_user_file {{ param_nginx_frontend['etc_dir'] }}/.htpasswd;
proxy_redirect off;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Host $http_host;
proxy_set_header Host $http_host;
proxy_connect_timeout 200;
proxy_send_timeout 200;
proxy_read_timeout 200;
send_timeout 200;
proxy_pass http://unix:{{ socket }};
}
location ~ ^(/login|/doLogin|/static|/slapgridResult|/isSRReady) {
proxy_redirect off;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Host $http_host;
proxy_set_header Host $http_host;
proxy_pass http://unix:{{ socket }};
}
location /shellinabox {
proxy_pass http://unix:{{ shellinabox_socket }}:/;
proxy_next_upstream error timeout invalid_header http_500 http_502 http_503 http_504;
auth_basic "Restricted";
auth_basic_user_file {{ param_nginx_frontend['etc_dir'] }}/.htpasswd;
proxy_redirect off;
proxy_buffering off;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Host $http_host;
}
}
}
{% set inst_parameter_dict = {} -%}
{% if slapparameter_dict is defined -%}
{% for key in slapparameter_dict.keys() -%}
{% if key.startswith('parameter-') -%}
{% do inst_parameter_dict.__setitem__(key[10:], slapparameter_dict.pop(key)) -%}
{% endif -%}
{% endfor -%}
{% endif -%}
<?xml version='1.0' encoding='utf-8'?>
<instance>
{% if slapparameter_dict is defined %}
{% for parameter_name in inst_parameter_dict.keys() %}
<parameter id="{{ parameter_name }}">{{ inst_parameter_dict[parameter_name] }}</parameter>
{% endfor %}
{% endif %}
</instance>
[buildout]
extends =
buildout.hash.cfg
../../component/bash/buildout.cfg
../../component/fish-shell/buildout.cfg
../../component/tmux/buildout.cfg
../../component/busybox/buildout.cfg
../../component/curl/buildout.cfg
../../component/dash/buildout.cfg
../../component/dcron/buildout.cfg
../../component/git/buildout.cfg
../../component/tig/buildout.cfg
../../component/logrotate/buildout.cfg
../../component/lxml-python/buildout.cfg
../../component/nano/buildout.cfg
../../component/nginx/buildout.cfg
../../component/openssh/buildout.cfg
../../component/mosh/buildout.cfg
../../component/rsync/buildout.cfg
../../component/pycurl/buildout.cfg
../../component/screen/buildout.cfg
../../component/shellinabox/buildout.cfg
../../component/vim/buildout.cfg
../../component/zip/buildout.cfg
../../stack/slapos.cfg
../../stack/flask.cfg
../../stack/resilient/buildout.cfg
../../stack/logrotate/buildout.cfg
../../stack/monitor/buildout.cfg
# make sure shared-part-list is available, even for old versions
# of slapos who do not set that.
shared-part-list =
# stacks are listed from most generic to most specific,
# to avoid versioning issues
common-parts =
slapos-command
slapos-cookbook
template
instance-runner-import
instance-runner-export
template-slapos-cfg
template-slapformat-definition.cfg
template-slapuser-script
# XXX: we have to manually add this for resilience
pbs-recipe-egg
parts =
${:common-parts}
[python]
part = python2.7
[template-base]
recipe = slapos.recipe.template
url = ${:_profile_base_location_}/${:filename}
[download-base]
recipe = slapos.recipe.build:download
url = ${:_profile_base_location_}/${:filename}
[template]
< = template-base
output = ${buildout:directory}/template.cfg
[template-runner]
< = template-base
output = ${buildout:directory}/template-runner.cfg.in
[template-runner-import-script]
< = download-base
[instance-runner-import]
< = download-base
[instance-runner-export]
< = download-base
[template-resilient]
< = download-base
[template_nginx_conf]
< = download-base
[template_httpd_conf]
< = download-base
[template_launcher]
< = download-base
[template-slapos-cfg]
< = download-base
[template-slapformat-definition.cfg]
< = download-base
[template-parameters]
< = download-base
[template-bash-profile]
< = download-base
[template-supervisord]
< = download-base
[template-listener-slapgrid]
< = download-base
[monitor-check-webrunner-internal-instance]
< = download-base
[template-resilient-software-release-information]
< = download-base
[template-slapuser-script]
< = download-base
[template-buildout-shared-part-list]
< = template-base
output = ${buildout:directory}/buildout-shared-part-list
[python-with-eggs]
recipe = zc.recipe.egg
interpreter = ${:_buildout_section_name_}
eggs =
${slapos-toolbox:eggs}
erp5.util
lock-file
slapos.recipe.build
slapos.toolbox[flask_auth]
gunicorn
# for gunicorn[gthread]
futures
supervisor
# BBB: eggs used as recipe should be kept otherwise sections depending
# on it can't be uninstalled
collective.recipe.shelloutput
scripts =
gunicorn
supervisord
supervisorctl
[versions]
Flask-Auth = 0.85
futures = 3.0.5
gunicorn = 19.10.0
prettytable = 0.7.2
pycurl = 7.43.0
{
"name": "Web Runner",
"description": "Web Runner",
"serialisation": "xml",
"software-type": {
"default": {
"title": "Default",
"description": "Standalone Runner",
"request": "instance-runner-input-schema.json",
"response": "instance-runner-output-schema.json",
"index": 0
},
"resilient": {
"title": "Resilient",
"description": "Resilient Runner",
"request": "instance-runner-resilient-input-schema.json",
"response": "instance-runner-output-schema.json",
"index": 1
}
}
}
# Beware, this file is automatically processed by slapgrid
# Do not modify it, your changes will be lost
# If you want to load your custom bash configuration, please use a .bashrc file
cd {{ workdir }}
export HOME={{- home }}
export PATH={{- path }}
export SHELL={{- shell }}
{%- if instance_name %}
export PROMPT_COMMAND='echo -en "\033]0;{{-instance_name}}\a"'
{% endif %}
export PS1="$ "
export TERMINFO={{- terminfo }}
if [ -f "$HOME/.bashrc" ] ; then
source $HOME/.bashrc
fi
# If not running interactively, don't do anything
case $- in
*i*) ;;
*) return;;
esac
# enable color support
if [ -x /usr/bin/dircolors ]; then
test -r ~/.dircolors && eval "$(dircolors -b ~/.dircolors)" || eval "$(dircolors -b)"
alias ls='ls --color=auto'
fi
# some more ls aliases
alias ll='ls -l'
alias la='ls -A'
alias l='ls -CF'
echo "Welcome to SlapOS slaprunner shell"
echo
#!{{ python_executable }}
import cgi
import cgitb
import ConfigParser
import os
import re
import subprocess
cgitb.enable(display=0, logdir="/tmp/cgi.log")
form = cgi.FieldStorage()
apache_file = "{{ apache_file }}"
config_file = "{{ config_cfg }}"
if not os.path.exists(config_file):
print "Your software does <b>not</b> embed 0-knowledge. \
This interface is useless in this case</body></html>"
exit(0)
parser = ConfigParser.ConfigParser()
parser.read(config_file)
if not parser.has_section("cors"):
parser.add_section("cors")
if not parser.has_option("cors", "cors-domain"):
parser.set("cors", "cors-domain", "")
if "cors-domain" in form:
parser.set("cors", "cors-domain", form["cors-domain"].value)
cors_domain_parameter = parser.get("cors", "cors-domain")
if cors_domain_parameter:
cors_domain_list = cors_domain_parameter.split()
cors_string = ""
for domain in cors_domain_list:
if cors_string:
cors_string += '|'
cors_string += re.escape(domain)
with open(apache_file, 'w') as file:
file.write('SetEnvIf Origin "^http(s)?://(.+\.)?(%s)$" origin_is=$0\n' % cors_string)
file.write('Header always set Access-Control-Allow-Origin %{origin_is}e env=origin_is')
subprocess.call('{{ httpd_graceful }}')
with open(config_file, 'w') as file:
parser.write(file)
print "<html><head>"
print "<link rel=\"stylesheet\" href=\"static/pure-min.css\">"
print "<link rel=\"stylesheet\" href=\"static/style.css\">"
print "</head><body>"
print "<h1>Cors Domains :</h1>"
print "<p>Enter domain names separated by space to authorized CORS on your dav storage for these domains</p>"
print "<form action=\"/index.cgi\" method=\"post\" class=\"pure-form-aligned\">"
print "<input type=\"hidden\" name=\"posting-script\" value=\"{{ pwd }}/{{ this_file }}\">"
print "<div class=\"pure-control-group\">"
print '<label for="cors-domain">CORS Domains</label>'
print '<input type="text" name="cors-domain" value="%s">' % cors_domain_parameter
print "</div>"
print "<div class=\"pure-controls\"><button type=\"submit\" class=\"pure-button \
pure-button-primary\">Save</button></div></form>"
print "</body></html>"
#!{{ python_executable }}
import datetime
import json
import sys
import xmlrpclib
from supervisor import childutils
def write_stdout(s):
sys.stdout.write(s)
sys.stdout.flush()
def write_stderr(s):
sys.stderr.write(s)
sys.stderr.flush()
def write_slapgrid_result(process):
server = xmlrpclib.Server("http://{{- supervisord['server'] -}}")
# Tuple of tuples containing 2 elements : process name and the path of its info file
json_files = dict([("slapgrid-sr", "{{- slaprunner['software_info_json'] -}}"),
("slapgrid-cp", "{{- slaprunner['instance_info_json'] -}}")])
info = server.supervisor.getProcessInfo(process)
result = dict()
result['last_build'] = datetime.datetime.fromtimestamp(info['stop']).strftime("%Y-%m-%d %H:%M:%S")
result['success'] = info['exitstatus']
open(json_files[process], 'w').write(json.dumps(result))
def main():
while 1:
headers, payload = writer.wait()
try:
processname = dict(x.split(':') for x in payload.split())['processname']
write_slapgrid_result(processname)
except:
pass
writer.ok()
if __name__ == '__main__':
writer = childutils.EventListenerProtocol()
main()
#!/usr/bin/python
import os
import subprocess
import sys
def runPromise(promise_path):
promise_relative_path = promise_path.replace(os.path.expanduser('~'), '~')
print 'Running promise %s...' % promise_relative_path
promise_process = subprocess.Popen(promise_path, stderr=subprocess.PIPE)
stdout, stderr = promise_process.communicate()
return_code = promise_process.returncode
if return_code == 0:
print 'Success.'
return True
else:
sys.stderr.write('Failure while running promise %s. %s\n' % (promise_relative_path, stderr))
def getPromisePathListFromPartitionPath(partition_path):
promise_directory_path = os.path.join(partition_path, 'etc/promise')
try:
promise_name_list = os.listdir(promise_directory_path)
return [os.path.join(promise_directory_path, promise_name) for promise_name in promise_name_list]
except OSError:
return []
def main():
# XXX hardcoded
partition_root_path = os.path.expanduser('~/srv/runner/instance')
success = True
for partition_name in os.listdir(partition_root_path):
partition_path = os.path.join(partition_root_path, partition_name)
for promise_path in getPromisePathListFromPartitionPath(partition_path):
result = runPromise(promise_path)
if not result:
success = False
if not success:
sys.exit(1)
if __name__ == '__main__':
main()
# Provides information related to the Webrunner Software Release to the
# takeover interface of the Resilient stack
def main():
return {
'Read the log from the importer': '<a href="${:resilient-log-url}">${:resilient-log-url}</a>',
}
#!{{ shell_binary }}
LC_ALL=C
export LC_ALL
umask 077
# Exit on any error, to prevent inconsistent backup
# Error on unset variable expansion
set -eu
# Redirect output to log
exec > >(tee -ai '{{ output_log_file }}')
exec 2>&1
echo -e "\n\n$0 run at : $(date)"
srv_directory='{{ directory["srv"] }}'
backup_directory='{{ directory["backup"] }}'
etc_directory='{{ directory["etc"] }}'
RESTORE_EXIT_CODE_FILE='{{ restore_exit_code_file }}'
RESTORE_ERROR_MESSAGE_FILE='{{ restore_error_message_file }}'
ERROR_MESSAGE=""
fail_with_exit_code () {
echo 1 > $RESTORE_EXIT_CODE_FILE
echo -e "Failure during step : $ERROR_MESSAGE" > $RESTORE_ERROR_MESSAGE_FILE
exit 1
}
trap fail_with_exit_code ERR
log_message () {
ERROR_MESSAGE=$1
echo -e $1
}
# Delete the error message file, to not keep it even after a successful build
rm "$RESTORE_ERROR_MESSAGE_FILE" || true
rsync () {
set -x
'{{ rsync_binary }}' -rlptgov --stats --safe-links --delete "$@"
set +x
}
log_message "Restoring WebRunner content..."
(
# XXX: code duplication with runner-export.sh.jinja2
path=$srv_directory/runner
backup_path=$backup_directory/runner/
cd "$backup_path"
if [ -d instance ]; then
# Concatenate the exclude file of each partition of webrunner
# to create a global exclude file.
# Also, ignore all buildout-managed files.
exclude=$({{ sys.executable }} - "$path" <<EOF
if 1:
import glob, errno, os, sys
sys.path[:0] = {{ repr(easy_install.buildout_and_setuptools_path) }}
from zc.buildout.configparser import parse
path = sys.argv[1]
def print_relative(path_list):
for p in path_list:
p = p.strip()
if p:
print(os.path.relpath(p, path))
print("*.sock")
print("*.socket")
print("*.pid")
print(".installed*.cfg")
for partition in glob.glob(path + "/instance/slappart*"):
try:
os.chdir(partition)
except OSError as e:
if e.errno != errno.ENOTDIR:
raise
continue
try:
with open("srv/exporter.exclude") as f:
exclude = f.readlines()
except IOError as e:
if e.errno != errno.ENOENT:
raise
else:
print_relative(exclude)
for installed in glob.glob(".installed*.cfg"):
try:
with open(installed) as f:
installed = parse(f, installed)
except IOError as e:
if e.errno != errno.ENOENT:
raise
else:
for section in installed.itervalues():
print_relative(section.get(
'__buildout_installed__', '').splitlines())
EOF
)
echo "$exclude" |rsync --exclude-from=- instance "$path"
fi
test -d project && rsync project "$path"
test -d public && rsync public "$path"
test -f proxy.db && rsync proxy.db "$path"
)
log_message "Restoring WebRunner config (etc directory)..."
(
cd "$backup_directory"/etc/
rsync config.json "$etc_directory"
# Hidden files are related to the webrunner's internals
cp -r .??* "$etc_directory"
)
# Invoke arbitrary script to perform specific restoration
# procedure.
runner_import_restore=$srv_directory/runner-import-restore
if [ -x "$runner_import_restore" ]; then
log_message "Running $runner_import_restore..."
"$srv_directory/runner-import-restore"
fi
# If no "etc/.project" neither "srv/runner/proxy.db", we can safely assume
# that there is no instance deployed on runner0
if [ ! -f "$etc_directory/.project" -a ! -f "$srv_directory/runner/proxy.db" ]; then
log_message "No Software Requested... Writing status file... End"
echo 0 > $RESTORE_EXIT_CODE_FILE
exit 0
fi
log_message "Updating slapproxy database..."
HOME='{{ directory["home"] }}'
# XXX Hardcoded
export PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
export MAKEFLAGS=-j4
SLAPOS='{{ directory["bin"] }}'/slapos
SQLITE3="{{ sqlite3_binary }}"
DATABASE="$HOME/srv/runner/proxy.db"
db_query () {
# Try opening locked tables for 5 seconds to prevent "database is locked" error
"$SQLITE3" "$DATABASE" <<EOF
.timeout 5000
$@
EOF
}
# If slapproxy database is empty then no software release was opened
if [ ! -s "$DATABASE" ]; then
log_message "Slapproxy database empty, no Software Requested... Writing status file... End"
echo 0 > $RESTORE_EXIT_CODE_FILE
exit 0
fi
# Slap proxy table contain version number, find the table name dynamically.
# This is known to work with version 11 or 12 of tables, but it will probably
# work with earlier versions as well.
DB_PARTITION_TABLE=$(db_query ".table partition__")
DB_PARTITION_NETWORK_TABLE=$(db_query ".table partition\_network__")
DB_SOFTWARE_TABLE=$(db_query ".table software__")
# Change slapproxy database to point instances to new software release
# XXX hardcoded
PARTITION=$(basename $HOME)
OLD_SOFTWARE_RELEASE=$(db_query "select software_release from $DB_PARTITION_TABLE where reference='slappart0';")
if [ "$OLD_SOFTWARE_RELEASE" == "" ];
then
echo "No instance configured"
echo 0 > $RESTORE_EXIT_CODE_FILE
exit 0
fi
SOFTWARE_RELEASE=$({{ sys.executable }} - $OLD_SOFTWARE_RELEASE $PARTITION <<EOF
if 1:
import os, re, sys
# We want to replace the last occurence only
old_software_release, partition = sys.argv[1], sys.argv[2]
for match in re.finditer("(slappart|test0-|s)[0-9][0-9]*", old_software_release):
start, end = match.start(), match.end()
print old_software_release[:start] + partition + old_software_release[end:]
EOF
)
db_query "update $DB_PARTITION_TABLE set software_release='$SOFTWARE_RELEASE' where software_release NOT NULL;"
db_query "update $DB_SOFTWARE_TABLE set url='$SOFTWARE_RELEASE' where url='$OLD_SOFTWARE_RELEASE';" || db_query "delete from $DB_SOFTWARE_TABLE where url='$OLD_SOFTWARE_RELEASE';"
# Change slapproxy database to have all instances stopped
db_query "update $DB_PARTITION_TABLE set requested_state='stopped';"
# Change slapproxy database to get correct IPs
IPV4='{{ ipv4 }}'
IPV6='{{ ipv6 }}'
db_query "update $DB_PARTITION_NETWORK_TABLE set address='$IPV4' where netmask='255.255.255.255';"
db_query "update $DB_PARTITION_NETWORK_TABLE set address='$IPV6' where netmask='ffff:ffff:ffff::';"
MASTERURL='http://{{ ipv4 }}:{{ proxy_port }}'
log_message "Removing old supervisord service description files..."
# XXX: Path hardcoded in slapos.core
rm '{{ instance_folder }}'/etc/supervisord.conf.d/* || true
SLAPOSCFG='{{ supervisord["slapos-cfg"] }}'
SLAPGRIDSRLOG='{{ supervisord["slapgrid-sr-log"] }}'
SLAPGRIDCPLOG='{{ supervisord["slapgrid-cp-log"] }}'
contain_software_release=0
SOFTWARE_RELEASES_COUNT=$(db_query "SELECT count(1) FROM $DB_SOFTWARE_TABLE WHERE url != '';")
if [ $SOFTWARE_RELEASES_COUNT -gt 0 ]; then
contain_software_release=1
fi
if [ $contain_software_release -eq 0 ]; then
log_message "No Software Release were deployed, so skip to continue..."
echo 0 > $RESTORE_EXIT_CODE_FILE
exit 0
fi
log_message "Building newest Software Release..."
"$SLAPOS" node software --cfg "$SLAPOSCFG" --all --master-url="$MASTERURL" --logfile "$SLAPGRIDSRLOG" >/dev/null 2>&1 ||
"$SLAPOS" node software --cfg "$SLAPOSCFG" --all --master-url="$MASTERURL" --logfile "$SLAPGRIDSRLOG" >/dev/null 2>&1 ||
"$SLAPOS" node software --cfg "$SLAPOSCFG" --all --master-url="$MASTERURL" --logfile "$SLAPGRIDSRLOG" >/dev/null 2>&1 ||
(tail -n 200 "$SLAPGRIDSRLOG" && false)
contain_instance=0
for folder in $srv_directory/runner/instance/slappart*/; do
if [ -f "$folder/buildout.cfg" ]; then
contain_instance=1
fi
done
# If instance do not contains template.cfg it means the user contains no instance.
# so it is safer to assume that he is using slaprunner for develop buildout rather them slapos.
if [ $contain_instance -eq 0 ]; then
log_message "None Instance were deployed with this software release, so skip to continue..."
echo 0 > $RESTORE_EXIT_CODE_FILE
exit 0
fi
# Remove defined scripts to force buildout to recreate them to have updated paths
rm "$srv_directory"/runner/instance/slappart*/srv/runner-import-restore || true
log_message "Fixing Instances as needed after import..."
# XXX hardcoded
"$SLAPOS" node instance --cfg "$SLAPOSCFG" --master-url=$MASTERURL --logfile "$SLAPGRIDCPLOG" >/dev/null 2>&1 ||
"$SLAPOS" node instance --cfg "$SLAPOSCFG" --master-url=$MASTERURL --logfile "$SLAPGRIDCPLOG" >/dev/null 2>&1 ||
"$SLAPOS" node instance --cfg "$SLAPOSCFG" --master-url=$MASTERURL --logfile "$SLAPGRIDCPLOG" >/dev/null 2>&1 ||
(tail -n 200 "$SLAPGRIDCPLOG" && false)
# Invoke defined scripts for each partition inside of slaprunner
log_message "Invoke custom import scripts defined by each instances..."
for partition in "$srv_directory"/runner/instance/slappart*/
do
script=$partition/srv/runner-import-restore
if [ -x "$script" ]; then
log_message "Running custom instance script : $script..."
"$script"
fi
done
# Change back slapproxy database to have all instances started
DB_PARTITION_TABLE=$(db_query ".table partition__")
log_message "Set instances as to start after takeover..."
db_query "update $DB_PARTITION_TABLE set requested_state='started';"
# Write exit code to an arbitrary file that will be checked by promise/monitor
log_message "Writing status file... End"
echo 0 > $RESTORE_EXIT_CODE_FILE
exit 0
[computer]
# Address example: 2001:67c:1254:27::1/ffff:ffff:ffff:ffff::/64
address = {{ slaprunner['ipv6'] }}/64
{% for partition_index in range(int(slaprunner['partition-amount'])) %}
[partition_{{ partition_index }}]
address = {{ slaprunner['ipv4'] }}/255.255.255.255 {{ slaprunner['ipv6'] }}/64
user = {{ partition_user }}
pathname = slappart{{ partition_index }}
network_interface =
{% endfor %}
#!/bin/sh
# run slapos command inside slaprunner with appropriate config
export SLAPOS_CONFIGURATION=${SLAPOS_CONFIGURATION-"{{ config_location }}"}
export SLAPOS_CLIENT_CONFIGURATION=${SLAPOS_CLIENT_CONFIGURATION-$SLAPOS_CONFIGURATION}
exec {{ slapos_python_file_location }} "$@"
[slapos]
software_root = {{ slaprunner['software_root'] }}
instance_root = {{ slaprunner['instance_root'] }}
shared_part_list =
{#- buildout_shared_part_list_dump is ${buildout:shared-part-list} rendered as a
template during software step.
Because it can contain new lines, it's not possible to use it directly when generating
buildout config files from buildout, because the newlines don't get indented and cause
the instance buildout to be invalid ini file.
So we had to dump it in a simple text file and now we parse again that text file. -#}
{%- for line in codecs.open(buildout_shared_part_list_dump).readlines() %}
{{ line.strip() }}
{%- endfor %}
{{ slaprunner['shared_root'] }}
master_url = http://{{ slaprunner['ipv4'] }}:{{ slaprunner['proxy_port'] }}
master_rest_url = http://{{ slaprunner['ipv4'] }}:{{ slaprunner['proxy_port'] }}/hateoas
computer_id = slaprunner
maximal_delay = 0
root_check = {{ slaprunner['root_check'] }}
forbid_supervisord_automatic_launch = true
pidfile_software = {{slaprunner['pidfile-software']}}
pidfile_instance = {{slaprunner['pidfile-instance']}}
[slapformat]
input_definition_file = {{ slaprunner['slapformat-definition.cfg'] }}
partition_amount = {{ slaprunner['partition-amount'] }}
alter_user = false
alter_network = false
create_tap = false
create_tun = false
computer_xml = {{ slaprunner['etc_dir'] }}/slapos.xml
[slaprunner]
slapos = {{ slaprunner['slapos'] }}
slapos_cfg = {{ slaprunner['slapos.cfg'] }}
slapproxy = {{ slaprunner['slapproxy'] }}
supervisor = {{ slaprunner['supervisor'] }}
supervisord_config = {{ slaprunner['supervisord_config'] }}
supervisord_server = {{ slaprunner['supervisord_server'] }}
runner_workdir = {{ slaprunner['working-directory'] }}
runner_host = {{ slaprunner['ipv4'] }}
runner_port = {{ slaprunner['runner_port'] }}
instance_monitoring_url = {{ slaprunner['instance-monitor-url'] }}
ipv4_address = {{ slaprunner['ipv4'] }}
ipv6_address = {{ slaprunner['ipv6'] }}
etc_dir = {{ slaprunner['etc_dir'] }}
run_dir = {{ slaprunner['run_dir'] }}
log_dir = {{ slaprunner['log_dir'] }}
console = {{ slaprunner['console'] }}
verbose = {{ slaprunner['verbose'] }}
debug = {{ slaprunner['debug'] }}
auto_deploy = {{ slaprunner['auto_deploy'] }}
auto_deploy_instance = {{ slaprunner['auto_deploy_instance'] }}
autorun = {{ slaprunner['autorun'] }}
knowledge0_cfg = {{ slaprunner['knowledge0_file'] }}
minishell_cwd_file = {{ slaprunner['minishell_cwd_file'] }}
minishell_history_file = {{ slaprunner['minishell_history_file'] }}
path = {{ slaprunner['path'] }}
instance_name = {{ slaprunner['instance_name'] }}
default_repository = {{ slaprunner['default_repository'] }}
default_repository_branch = {{ slaprunner['default_repository_branch'] }}
[slapproxy]
host = {{ slaprunner['ipv4'] }}
port = {{ slaprunner['proxy_port'] }}
database_uri = {{ slaprunner['proxy_database'] }}
[gitclient]
git = {{ slaprunner['git-binary'] }}
[sshkeys_authority]
public_key = {{ slaprunner['public_key'] }}
[unix_http_server]
file = {{ supervisord['socket_path'] }}
[supervisorctl]
serverurl = {{ supervisord['socket_name'] }}
[inet_http_server]
port = {{ supervisord['server'] }}
[supervisord]
logfile = {{ supervisord['logfile'] }}
pidfile = {{ supervisord['pidfile'] }}
[rpcinterface:supervisor]
supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface
[program:{{- supervisord['slapgrid-sr'] -}}]
command = {{ supervisord['slapgrid-sr-command'] }}
process_name = {{ supervisord['slapgrid-sr'] }}
numprocs = {{ supervisord['numprocs'] }}
autostart = {{ supervisord['autostart'] }}
exitcodes = {{ supervisord['exitcodes'] }}
startretries = {{ supervisord['slapgrid-sr-startretries'] }}
startsecs = {{ supervisord['startsecs'] }}
autorestart = {{ supervisord['autorestart'] }}
stdout_logfile = {{ supervisord['no_logfile'] }}
stderr_logfile = {{ supervisord['no_logfile'] }}
directory = {{ supervisord['directory'] }}
{# how many parallel build jobs to spawn when compiling software -#}
{% set njobs = max(1, (multiprocessing.cpu_count() // int(slapparameter_dict.get('cpu-usage-ratio', 4)))) -%}
environment = PATH="{{- supervisord['path'] -}}",MAKEFLAGS="-j{{ njobs }}",NPY_NUM_BUILD_JOBS="{{ njobs }}",BUNDLE_JOBS="{{ njobs }}"
[program:{{- supervisord['slapgrid-cp'] -}}]
command = {{ supervisord['slapgrid-cp-command'] }}
process_name = {{ supervisord['slapgrid-cp'] }}
numprocs = {{ supervisord['numprocs'] }}
autostart = {{ supervisord['autostart'] }}
exitcodes = {{ supervisord['exitcodes'] }}
startretries = {{ supervisord['slapgrid-cp-startretries'] }}
startsecs = {{ supervisord['startsecs'] }}
autorestart = {{ supervisord['autorestart'] }}
stdout_logfile = {{ supervisord['no_logfile'] }}
stderr_logfile = {{ supervisord['no_logfile'] }}
directory = {{ supervisord['directory'] }}
environment = PATH="{{- supervisord['path'] -}}"
[program:{{- supervisord['slapformat'] -}}]
command = {{ supervisord['slapformat-command'] }}
process_name = {{ supervisord['slapformat'] }}
numprocs = {{ supervisord['numprocs'] }}
autostart = {{ supervisord['autostart'] }}
exitcodes = {{ supervisord['exitcodes'] }}
startretries = {{ supervisord['slapformat-startretries'] }}
startsecs = {{ supervisord['startsecs'] }}
autorestart = {{ supervisord['autorestart'] }}
stdout_logfile = {{ supervisord['no_logfile'] }}
stderr_logfile = {{ supervisord['no_logfile'] }}
directory = {{ supervisord['directory'] }}
environment = PATH="{{- supervisord['path'] -}}"
[program:{{- supervisord['slapboot'] -}}]
command = {{ supervisord['slapboot-command'] }}
process_name = {{ supervisord['slapboot'] }}
numprocs = {{ supervisord['numprocs'] }}
autostart = {{ supervisord['autostart'] }}
exitcodes = {{ supervisord['exitcodes'] }}
startretries = {{ supervisord['slapboot-startretries'] }}
startsecs = {{ supervisord['startsecs'] }}
autorestart = {{ supervisord['autorestart'] }}
stdout_logfile = {{ supervisord['no_logfile'] }}
stderr_logfile = {{ supervisord['no_logfile'] }}
directory = {{ supervisord['directory'] }}
environment = PATH="{{- supervisord['path'] -}}"
[program:{{- supervisord['slapbang'] -}}]
command = {{ supervisord['slapbang-command'] }}
process_name = {{ supervisord['slapbang'] }}
numprocs = {{ supervisord['numprocs'] }}
autostart = {{ supervisord['autostart'] }}
exitcodes = {{ supervisord['exitcodes'] }}
startretries = {{ supervisord['slapbang-startretries'] }}
startsecs = {{ supervisord['startsecs'] }}
autorestart = {{ supervisord['autorestart'] }}
stdout_logfile = {{ supervisord['no_logfile'] }}
stderr_logfile = {{ supervisord['no_logfile'] }}
directory = {{ supervisord['directory'] }}
environment = PATH="{{- supervisord['path'] -}}"
[program:{{- supervisord['slapproxy'] -}}]
command = {{ supervisord['slapproxy-command'] }}
process_name = {{ supervisord['slapproxy'] }}
numprocs = {{ supervisord['numprocs'] }}
autostart = {{ supervisord['slapproxy-autostart'] }}
startsecs = {{ supervisord['slapproxy-startsecs'] }}
exitcodes = {{ supervisord['exitcodes'] }}
autorestart = {{ supervisord['slapproxy-autorestart'] }}
stdout_logfile = {{ supervisord['no_logfile'] }}
stderr_logfile = {{ supervisord['no_logfile'] }}
directory = {{ supervisord['directory'] }}
environment = PATH="{{- supervisord['path'] -}}"
[eventlistener:listener-slapgrid]
command = {{ listener_slapgrid }}
events = PROCESS_STATE_EXITED
stdout_logfile = {{ supervisord['no_logfile'] }}
stderr_logfile = {{ supervisord['stderr_logfile'] }}
Tests for slaprunner Software Release
##############################################################################
#
# Copyright (c) 2018 Nexedi SA and Contributors. All Rights Reserved.
#
# WARNING: This program as such is intended to be used by professional
# programmers who take the whole responsibility of assessing all potential
# consequences resulting from its eventual inadequacies and bugs
# End users who are looking for a ready-to-use solution with commercial
# guarantees and support are strongly adviced to contract a Free Software
# Service Company
#
# This program is Free Software; you can redistribute it and/or
# modify it under the terms of the GNU General Public License
# as published by the Free Software Foundation; either version 3
# of the License, or (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.
#
##############################################################################
from setuptools import setup, find_packages
version = '0.0.1.dev0'
name = 'slapos.test.slaprunner'
with open("README.md") as f:
long_description = f.read()
setup(name=name,
version=version,
description="Test for SlapOS' slaprunner",
long_description=long_description,
long_description_content_type='text/markdown',
maintainer="Nexedi",
maintainer_email="info@nexedi.com",
url="https://lab.nexedi.com/nexedi/slapos",
packages=find_packages(),
install_requires=[
'slapos.core',
'slapos.cookbook',
'slapos.libnetworkcache',
'erp5.util',
'supervisor',
'psutil',
'paramiko',
'requests',
],
zip_safe=True,
test_suite='test',
)
##############################################################################
#
# Copyright (c) 2018 Nexedi SA and Contributors. All Rights Reserved.
#
# WARNING: This program as such is intended to be used by professional
# programmers who take the whole responsibility of assessing all potential
# consequences resulting from its eventual inadequacies and bugs
# End users who are looking for a ready-to-use solution with commercial
# guarantees and support are strongly adviced to contract a Free Software
# Service Company
#
# This program is Free Software; you can redistribute it and/or
# modify it under the terms of the GNU General Public License
# as published by the Free Software Foundation; either version 3
# of the License, or (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.
#
##############################################################################
import os
import unittest
import paramiko
import contextlib
import base64
import hashlib
import subprocess
import json
import time
from urllib.parse import urlparse
from urllib.parse import quote
from urllib.parse import urljoin
from configparser import ConfigParser
import requests
from slapos.recipe.librecipe import generateHashFromFiles
from slapos.testing.testcase import makeModuleSetUpAndTestCaseClass
from slapos.util import bytes2str
setUpModule, SlapOSInstanceTestCase = makeModuleSetUpAndTestCaseClass(
os.path.abspath(
os.path.join(os.path.dirname(__file__), '..', 'software.cfg')))
class SlaprunnerTestCase(SlapOSInstanceTestCase):
# Slaprunner uses unix sockets, so it needs short paths.
__partition_reference__ = 's'
def _openSoftwareRelease(self, software_release="erp5testnode/testsuite/dummy"):
parameter_dict = self.computer_partition.getConnectionParameterDict()
url = "%s/setCurrentProject" % parameter_dict['url']
data = {
"path": "workspace/slapos/software/%s" % software_release,
}
resp = self._postToSlaprunner(url, data)
self.assertEqual(requests.codes.ok, resp.status_code)
self.assertNotEqual(json.loads(resp.text)['code'], 0,
'Unexpecting result in call to setCurrentProject: %s' % resp.text)
def _buildSoftwareRelease(self):
parameter_dict = self.computer_partition.getConnectionParameterDict()
url = "%s/runSoftwareProfile" % parameter_dict['url']
resp = self._postToSlaprunner(url, {})
self.assertEqual(requests.codes.ok, resp.status_code)
self.assertEqual(json.loads(resp.text)['result'], True,
'Unexpecting result in call to runSoftwareProfile: %s' % resp.text)
def _deployInstance(self):
parameter_dict = self.computer_partition.getConnectionParameterDict()
url = "%s/runInstanceProfile" % parameter_dict['url']
resp = self._postToSlaprunner(url, {})
self.assertEqual(requests.codes.ok, resp.status_code)
self.assertEqual(json.loads(resp.text)['result'], True,
'Unexpecting result in call to runSoftwareProfile: %s' % resp.text)
def _gitClone(self):
parameter_dict = self.computer_partition.getConnectionParameterDict()
url = "%s/cloneRepository" % parameter_dict['url']
data = {
"repo": "https://lab.nexedi.com/nexedi/slapos.git",
"name": "workspace/slapos",
"email": "slapos@slapos.org",
"user": "slapos"
}
resp = self._postToSlaprunner(url, data)
d = json.loads(resp.text)
if d['code'] == 0:
return "OK"
def _isSoftwareReleaseReady(self):
parameter_dict = self.computer_partition.getConnectionParameterDict()
url = "%s/isSRReady" % parameter_dict['url']
resp = self._getFromSlaprunner(url)
if requests.codes.ok != resp.status_code:
return -1
return resp.text
def _waitForSoftwareBuild(self, limit=5000):
status = self._isSoftwareReleaseReady()
while limit > 0 and status != "1":
status = self._isSoftwareReleaseReady()
limit -= 1
if status == '0':
self.logger.debug("Software release is Failing to Build. Sleeping...")
else:
self.logger.debug('Software is still building. Sleeping...')
time.sleep(20)
def _waitForInstanceDeploy(self):
parameter_dict = self.computer_partition.getConnectionParameterDict()
url = "%s/slapgridResult" % parameter_dict['url']
data = {
"position": 0,
"log": ""
}
while True:
time.sleep(25)
resp = self._postToSlaprunner(url, data)
if requests.codes.ok != resp.status_code:
continue
if json.loads(resp.text)["instance"]["state"] is False:
break
self.logger.info('Buildout is still running. Sleeping....')
self.logger.info("Instance has been deployed.")
def _getFromSlaprunner(self, url):
parameter_dict = self.computer_partition.getConnectionParameterDict()
return requests.get(
url,
verify=False,
auth=(parameter_dict['init-user'], parameter_dict['init-password']))
def _postToSlaprunner(self, url, data):
parameter_dict = self.computer_partition.getConnectionParameterDict()
return requests.post(
url,
verify=False,
data=data,
auth=(parameter_dict['init-user'], parameter_dict['init-password']))
def _getFileContent(self, relative_path):
parameter_dict = self.computer_partition.getConnectionParameterDict()
url = "%s/getFileContent" % parameter_dict['url']
data = {
"file": relative_path
}
resp = self._postToSlaprunner(url, data)
self.assertEqual(requests.codes.ok, resp.status_code)
self.assertNotEqual(json.loads(resp.text)['code'], 0,
'Unexpecting result in call to getFileContent: %s' % resp.text)
return json.loads(resp.text)["result"]
def _waitForCloneToBeReadyForTakeover(self, scope="runner-1", limit=500):
parameter_dict = self.computer_partition.getConnectionParameterDict()
takeover_url = parameter_dict["takeover-%s-url" % scope]
def getTakeoverPageContent():
resp = requests.get(takeover_url, verify=True)
self.assertEqual(requests.codes.ok, resp.status_code)
return resp.text
takeover_page_content = getTakeoverPageContent()
while "<b>Last valid backup:</b> No backup downloaded yet, takeover should not happen now." in takeover_page_content:
time.sleep(10)
if limit < 0:
raise Exception("Timeout: No valid Backup")
takeover_page_content = getTakeoverPageContent()
limit -= 1
while "<b>Importer script(s) of backup in progress:</b> True" in takeover_page_content:
time.sleep(10)
if limit < 0:
raise Exception("Timeout: Backup still in progress")
takeover_page_content = getTakeoverPageContent()
limit -= 1
def _doTakeover(self, scope="runner-1"):
parameter_dict = self.computer_partition.getConnectionParameterDict()
takeover_url = parameter_dict["takeover-%s-url" % scope]
takeover_password = parameter_dict["takeover-%s-password" % scope]
resp = requests.get(
f"{takeover_url}?password={takeover_password}",
verify=True)
self.assertEqual(requests.codes.ok, resp.status_code)
self.assertNotIn("Error", resp.text,
"An Error occured: %s" % resp.text)
self.assertIn("Success", resp.text,
"An Success not in %s" % resp.text)
return resp.text
class TestWebRunnerBasicUsage(SlaprunnerTestCase):
@classmethod
def getInstanceParameterDict(cls):
return {
'auto-deploy': 'true',
'software-root': os.path.join(cls.slap._instance_root, "..", "soft"),
'buildout-shared-folder': os.path.join(cls.slap._instance_root, "..", "shared"),
"slapos-reference": 'slaprunner-basic-test-resiliency'
}
def test_open_software_release(self):
self._openSoftwareRelease()
def test_git_clone(self):
self._gitClone()
@unittest.skip('Skip as _getFileContent dont work for now')
def test_basic_usage(self):
self._openSoftwareRelease()
self._buildSoftwareRelease()
self._waitForSoftwareBuild()
self._deployInstance()
self._waitForInstanceDeploy()
result = self._getFileContent(
"instance_root/slappart0/var/log/log.log")
self.assertTrue(result.startswith("Hello"),
result)
class TestWebRunnerAutorun(SlaprunnerTestCase):
@classmethod
def getInstanceParameterDict(cls):
return {
# Auto deploy is required for the isSRReady works.
'auto-deploy': 'true',
'autorun': 'true',
'software-root': os.path.join(cls.slap._instance_root, "..", "soft"),
'buildout-shared-folder': os.path.join(cls.slap._instance_root, "..", "shared"),
'slapos-software': 'software/erp5testnode/testsuite/dummy',
# XXX HACK!
"slapos-reference": 'slaprunner-basic-test-resiliency'
}
@unittest.skip('Skip as _getFileContent dont work for now')
def test_basic_usage(self):
self._openSoftwareRelease()
self._waitForSoftwareBuild()
self._waitForSoftwareBuild()
self._waitForInstanceDeploy()
self._waitForInstanceDeploy()
result = self._getFileContent(
"instance_root/slappart0/var/log/log.log")
self.assertTrue(result.startswith("Hello"), result)
class TestWeb(SlaprunnerTestCase):
def test_slaprunner(self):
# slaprunner main interface is password protected
parameter_dict = self.computer_partition.getConnectionParameterDict()
url = parameter_dict['url']
resp = requests.get(url, verify=False)
self.assertEqual(requests.codes.unauthorized, resp.status_code)
resp = requests.get(
url,
verify=False,
auth=(parameter_dict['init-user'], parameter_dict['init-password']))
self.assertEqual(requests.codes.ok, resp.status_code)
self.assertIn('SlapOS', resp.text)
def test_slaprunner_redirects(self):
# redirects also work as expected. In this test we visit stopAllPartition
# which should redirect to inspectInstance
parameter_dict = self.computer_partition.getConnectionParameterDict()
url = parameter_dict['url']
resp = requests.get(
urljoin(url, '/stopAllPartition'),
verify=False,
auth=(parameter_dict['init-user'], parameter_dict['init-password']))
self.assertEqual(resp.status_code, requests.codes.ok)
self.assertEqual(resp.url, urljoin(url, '/inspectInstance'))
# this also works behind a frontend
resp = requests.get(
urljoin(url, '/stopAllPartition'),
verify=False,
allow_redirects=False,
headers={'Host': 'example.com:1234'},
auth=(parameter_dict['init-user'], parameter_dict['init-password']))
self.assertEqual(resp.status_code, requests.codes.found)
self.assertEqual(resp.headers['Location'], 'https://example.com:1234/inspectInstance')
def test_shellinabox(self):
# shellinabox exists at /shellinabox and is password protected
parameter_dict = self.computer_partition.getConnectionParameterDict()
url = urljoin(parameter_dict['url'], '/shellinabox')
resp = requests.get(url, verify=False)
self.assertEqual(requests.codes.unauthorized, resp.status_code)
resp = requests.get(
url,
verify=False,
auth=(parameter_dict['init-user'], parameter_dict['init-password']))
self.assertEqual(requests.codes.ok, resp.status_code)
self.assertIn('ShellInABox', resp.text)
self.assertNotIn('SlapOS', resp.text)
def test_public_url(self):
# ~/srv/runner/public/ is served over http
parameter_dict = self.computer_partition.getConnectionParameterDict()
public_url = parameter_dict['public-url']
hello_file = os.path.join(
self.computer_partition_root_path,
'srv',
'runner',
'public',
'hello.html')
self.addCleanup(os.remove, hello_file)
with open(hello_file, 'w') as f:
f.write('<b>Hello</b>')
index = requests.get(public_url, verify=False)
self.assertEqual(requests.codes.ok, index.status_code)
self.assertIn('hello.html', index.text)
hello = requests.get(urljoin(public_url, 'hello.html'), verify=False)
self.assertEqual(requests.codes.ok, hello.status_code)
self.assertIn('<b>Hello</b>', hello.text)
class TestSSH(SlaprunnerTestCase):
@classmethod
def getInstanceParameterDict(cls):
cls.ssh_key_list = [paramiko.ECDSAKey.generate(bits=384) for i in range(2)]
return {
'user-authorized-key': 'ecdsa-sha2-nistp384 {}\necdsa-sha2-nistp384 {}'.format(
*[key.get_base64() for key in cls.ssh_key_list]
)
}
def test_connect(self):
parameter_dict = self.computer_partition.getConnectionParameterDict()
ssh_url = parameter_dict['ssh-url']
parsed = urlparse(ssh_url)
self.assertEqual('ssh', parsed.scheme)
# username contain a fingerprint (only, so we simplify the parsing)
#
# relevant parts of the grammar defined in
# https://tools.ietf.org/id/draft-salowey-secsh-uri-00.html
#
# ssh-info = [ userinfo ] [";" c-param *("," c-param)]
# c-param = paramname "=" paramvalue
ssh_info = parsed.username
username, fingerprint_from_url = ssh_info.split(';fingerprint=')
client = paramiko.SSHClient()
self.assertTrue(fingerprint_from_url.startswith('ssh-rsa-'), fingerprint_from_url)
fingerprint_from_url = fingerprint_from_url[len('ssh-rsa-'):]
class KeyPolicy:
"""Accept server key and keep it in self.key for inspection
"""
def missing_host_key(self, client, hostname, key):
self.key = key
key_policy = KeyPolicy()
client.set_missing_host_key_policy(key_policy)
for ssh_key in self.ssh_key_list:
with contextlib.closing(client):
client.connect(
username=username,
hostname=parsed.hostname,
port=parsed.port,
pkey=ssh_key,
)
# Check fingerprint from server matches the published one.
# Paramiko does not allow to get the fingerprint as SHA256 easily yet
# https://github.com/paramiko/paramiko/pull/1103
self.assertEqual(
fingerprint_from_url,
quote(
# base64 encoded fingerprint adds an extra = at the end
base64.b64encode(
hashlib.sha256(key_policy.key.asbytes()).digest())[:-1],
# also encode /
safe=''))
# Check shell is usable
channel = client.invoke_shell()
channel.settimeout(30)
received = ''
while True:
r = bytes2str(channel.recv(1024))
self.logger.debug("received >%s<", r)
if not r:
break
received += r
if 'slaprunner shell' in received:
break
self.assertIn("Welcome to SlapOS slaprunner shell", received)
# simple commands can also be executed ( this would be like `ssh bash -c 'pwd'` )
self.assertEqual(
self.computer_partition_root_path,
bytes2str(client.exec_command("pwd")[1].read(1000)).strip())
class TestSlapOS(SlaprunnerTestCase):
def test_slapos_command(self):
# in ~/bin/slapos there is a wrapper setting configuration to use slapos from
# the web runner.
slapos = os.path.join(self.computer_partition_root_path, 'bin', 'slapos')
# ensure the node is formatted
subprocess.check_call((slapos, 'node', 'format', '--now'), env={})
proxy_show_output = subprocess.check_output((slapos, 'proxy', 'show'), env={})
self.assertIn(b'slaprunner', proxy_show_output)
# check hateoas cli support
computer_list_output = subprocess.check_output((slapos, 'computer', 'list'), env={})
self.assertIn(b'slaprunner', computer_list_output)
def test_shared_part_list(self):
# this slapos used shared_part_list
cfg_parser = ConfigParser()
with open(os.path.join(self.computer_partition_root_path,
'etc',
'slapos.cfg')) as f:
cfg_parser.readfp(f)
shared_part_list = cfg_parser.get('slapos', 'shared_part_list').splitlines()
# web runner own shared parts. Note that there is intentionnaly a double
# slash in this path, because slaprunner has double slash in paths since
# early releases, including for the path of slapos repository that will be
# used to develop and install software. If we fix this duplication, then
# the URL of installed software will be different and it will get a different
# hash and be reinstalled. To prevent this, we keep that // between srv and runner.
self.assertEqual(
'{}/srv//runner//shared'.format(self.computer_partition_root_path.rstrip('/')),
shared_part_list[-1])
# shared parts from outer slapos
outer_shared_part_list = os.getenv('SLAPOS_TEST_SHARED_PART_LIST',
'').split(os.pathsep)
for outer_shared_part in outer_shared_part_list:
self.assertIn(outer_shared_part, shared_part_list)
class ServicesTestCase(SlaprunnerTestCase):
def test_hashes(self):
hash_files = [
'software_release/buildout.cfg',
]
expected_process_names = [
'slaprunner-supervisord-{hash}-on-watch',
'runner-sshd-{hash}-on-watch',
'slaprunner-httpd-{hash}-on-watch',
'gunicorn-{hash}-on-watch',
'nginx-frontend-{hash}-on-watch',
'certificate_authority-{hash}-on-watch',
'shellinaboxd-{hash}-on-watch',
'supervisord-{hash}-on-watch',
]
with self.slap.instance_supervisor_rpc as supervisor:
process_names = [
process['name'] for process in supervisor.getAllProcessInfo()
]
hash_files = [os.path.join(self.computer_partition_root_path, path)
for path in hash_files]
for name in expected_process_names:
h = generateHashFromFiles(hash_files)
expected_process_name = name.format(hash=h)
self.assertIn(expected_process_name, process_names)
class TestCustomFrontend(SlaprunnerTestCase):
@classmethod
def getInstanceParameterDict(cls):
return {
'custom-frontend-backend-url': 'https://www.erp5.com',
'custom-frontend-backend-type': 'redirect',
}
def test(self):
parameter_dict = self.computer_partition.getConnectionParameterDict()
# slapproxy returns the backend URL when requesting a slave frontend
self.assertEqual(
parameter_dict['custom-frontend-url'],
'https://www.erp5.com')
class TestResilientInstance(SlaprunnerTestCase):
instance_max_retry = 20
@classmethod
def getInstanceSoftwareType(cls):
return 'resilient'
def test(self):
# just check that keys returned on requested partition are for resilient
self.assertSetEqual(
set(self.computer_partition.getConnectionParameterDict().keys()),
{
'backend-url',
'feed-url-runner-1-pull',
'feed-url-runner-1-push',
'init-password',
'init-user',
'monitor-base-url',
'monitor-setup-url',
'public-url',
'ssh-command',
'takeover-runner-1-password',
'takeover-runner-1-url',
'url',
'webdav-url'})
class TestResilientCustomFrontend(TestCustomFrontend):
instance_max_retry = 20
@classmethod
def getInstanceSoftwareType(cls):
return 'resilient'
class TestResilientWebInstance(TestWeb):
instance_max_retry = 20
@classmethod
def getInstanceSoftwareType(cls):
return 'resilient'
def test_public_url(self):
pass # Disable until we can write on runner0 rather them
# on root partition
class TestResilientWebrunnerBasicUsage(TestWebRunnerBasicUsage):
instance_max_retry = 20
@classmethod
def getInstanceSoftwareType(cls):
return 'resilient'
class TestResilientWebrunnerAutorun(TestWebRunnerAutorun):
instance_max_retry = 20
@classmethod
def getInstanceSoftwareType(cls):
return 'resilient'
class TestResilientDummyInstance(SlaprunnerTestCase):
instance_max_retry = 20
@classmethod
def getInstanceSoftwareType(cls):
return 'resilient'
@classmethod
def getInstanceParameterDict(cls):
return {
'resiliency-backup-periodicity': '*/6 * * * *',
'auto-deploy-instance': 'false',
'software-root': os.path.join(cls.slap._instance_root, "..", "soft"),
'buildout-shared-folder': os.path.join(cls.slap._instance_root, "..", "shared"),
'auto-deploy': 'true',
# XXX HACK!
"slapos-reference": 'slaprunner-erp5-resiliency',
"slapos-httpd-port": '9687'
}
@unittest.skip('Skip as _getFileContent dont work for now')
def test_basic_resilience(self):
self._openSoftwareRelease()
self._buildSoftwareRelease()
self._waitForSoftwareBuild()
self._deployInstance()
self._waitForInstanceDeploy()
result = self._getFileContent(
"instance_root/slappart0/var/log/log.log")
self.assertTrue(result.startswith("Hello"), result)
# We should ensure here that the resilience was indeed
# Propagates and test succeeded.
time.sleep(900)
self._waitForCloneToBeReadyForTakeover()
self._doTakeover()
self.slap.waitForInstance(20)
previous_computer_partition = self.computer_partition
self.computer_partition = self.requestDefaultInstance()
result_after = self._getFileContent(
"instance_root/slappart0/var/log/log.log")
self.assertTrue(result_after.startswith("Hello"), result_after)
self.assertIn(result, result_after,
f"{result} not in {result_after}")
...@@ -78,8 +78,8 @@ If you have sub-instances, you should collect the base monitor url from all inst ...@@ -78,8 +78,8 @@ If you have sub-instances, you should collect the base monitor url from all inst
Also, all monitors of the sub instances need to have same password as the password of the root instance monitor. Also, all monitors of the sub instances need to have same password as the password of the root instance monitor.
NB: You should use double $ (ex: $${monitor-template:output}) instead of one $ in your instance template file if it's not a jinja template. See: NB: You should use double $ (ex: $${monitor-template:output}) instead of one $ in your instance template file if it's not a jinja template. See:
- Jinja template file exemple, use one $: https://lab.nexedi.com/nexedi/slapos/blob/master/software/slaprunner/instance-resilient-test.cfg.jinja2 - Jinja template file exemple, use one $: https://lab.nexedi.com/nexedi/slapos/blob/master/software/theia/instance-resilient.cfg.jinja
- Non Jinja template file, use $$: https://lab.nexedi.com/nexedi/slapos/blob/master/software/slaprunner/instance.cfg - Non Jinja template file, use $$: https://lab.nexedi.com/nexedi/slapos/blob/master/software/theia/instance.cfg.in
Add a promise Add a promise
------------- -------------
......
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment