Skip to content
Projects
Groups
Snippets
Help
Loading...
Help
Support
Keyboard shortcuts
?
Submit feedback
Contribute to GitLab
Sign in / Register
Toggle navigation
slapos
Project overview
Project overview
Details
Activity
Releases
Repository
Repository
Files
Commits
Branches
Tags
Contributors
Graph
Compare
Issues
0
Issues
0
List
Boards
Labels
Milestones
Merge Requests
0
Merge Requests
0
CI / CD
CI / CD
Pipelines
Jobs
Schedules
Analytics
Analytics
CI / CD
Repository
Value Stream
Wiki
Wiki
Snippets
Snippets
Members
Members
Collapse sidebar
Close sidebar
Activity
Graph
Create a new issue
Jobs
Commits
Issue Boards
Open sidebar
Łukasz Nowak
slapos
Commits
bce9b4af
Commit
bce9b4af
authored
Sep 28, 2021
by
Łukasz Nowak
Browse files
Options
Browse Files
Download
Email Patches
Plain Diff
XXX rapid-cdn: Implement simple and working frontend support
Checkpoint: returning kedifa-caucase-url XXX: TODO
parent
13b24cb7
Changes
11
Hide whitespace changes
Inline
Side-by-side
Showing
11 changed files
with
774 additions
and
2 deletions
+774
-2
software/rapid-cdn/buildout.hash.cfg
software/rapid-cdn/buildout.hash.cfg
+41
-0
software/rapid-cdn/instance-common.cfg.jinja2
software/rapid-cdn/instance-common.cfg.jinja2
+49
-0
software/rapid-cdn/instance-rapid-cdn-frontend.cfg.jinja2
software/rapid-cdn/instance-rapid-cdn-frontend.cfg.jinja2
+199
-0
software/rapid-cdn/instance-rapid-cdn-human.cfg.jinja2
software/rapid-cdn/instance-rapid-cdn-human.cfg.jinja2
+4
-0
software/rapid-cdn/instance-rapid-cdn-kedifa.cfg.jinja2
software/rapid-cdn/instance-rapid-cdn-kedifa.cfg.jinja2
+163
-0
software/rapid-cdn/instance-rapid-cdn-log-aggregator.cfg.jinja2
...re/rapid-cdn/instance-rapid-cdn-log-aggregator.cfg.jinja2
+4
-0
software/rapid-cdn/instance-rapid-cdn-pop.cfg.jinja2
software/rapid-cdn/instance-rapid-cdn-pop.cfg.jinja2
+41
-0
software/rapid-cdn/instance.cfg.jinja2
software/rapid-cdn/instance.cfg.jinja2
+61
-0
software/rapid-cdn/software.cfg
software/rapid-cdn/software.cfg
+100
-0
software/rapid-cdn/test/setup.py
software/rapid-cdn/test/setup.py
+2
-0
software/rapid-cdn/test/test.py
software/rapid-cdn/test/test.py
+110
-2
No files found.
software/rapid-cdn/buildout.hash.cfg
0 → 100644
View file @
bce9b4af
# THIS IS NOT A BUILDOUT FILE, despite purposedly using a compatible syntax.
# The only allowed lines here are (regexes):
# - "^#" comments, copied verbatim
# - "^[" section beginings, copied verbatim
# - lines containing an "=" sign which must fit in the following categorie.
# - "^\s*filename\s*=\s*path\s*$" where "path" is relative to this file
# Copied verbatim.
# - "^\s*hashtype\s*=.*" where "hashtype" is one of the values supported
# by the re-generation script.
# Re-generated.
# - other lines are copied verbatim
# Substitution (${...:...}), extension ([buildout] extends = ...) and
# section inheritance (< = ...) are NOT supported (but you should really
# not need these here).
[template]
filename = instance.cfg.jinja2
md5sum = 23b50d7f04b201d55b9e27aa8dc3ba87
[template-common]
filename = instance-common.cfg.jinja2
md5sum = c3a54ac864508ecb693c16cdec06671e
[template-rapid-cdn-human]
filename = instance-rapid-cdn-human.cfg.jinja2
md5sum = 6a57e7727c8d258f5ae81bdbb86328cb
[template-rapid-cdn-pop]
filename = instance-rapid-cdn-pop.cfg.jinja2
md5sum = 41738610be11b3af6b9961b91629a1a1
[template-rapid-cdn-kedifa]
filename = instance-rapid-cdn-kedifa.cfg.jinja2
md5sum = 94b4c24d71def7ab08d88682a9814179
[template-rapid-cdn-log-aggregator]
filename = instance-rapid-cdn-log-aggregator.cfg.jinja2
md5sum = 6a57e7727c8d258f5ae81bdbb86328cb
[template-rapid-cdn-frontend]
filename = instance-rapid-cdn-frontend.cfg.jinja2
md5sum = c9afbc178c6523e8a518faadd576dc92
software/rapid-cdn/instance-common.cfg.jinja2
0 → 100644
View file @
bce9b4af
[buildout]
eggs-directory = {{ software_parameter_dict['eggs-directory'] }}
develop-eggs-directory = {{ software_parameter_dict['develop-eggs-directory'] }}
offline = true
[kedifa-config]
# shared kedifa configuration parameters for the whole cluster
marker = __NotReadyYet__
[csr]
recipe = plone.recipe.command
organization = ${slap-configuration:root-instance-title}
command =
if [ ! -f ${:template-csr} ] && [ ! -f ${:key} ] ; then
/bin/bash -c '{{ software_parameter_dict['openssl'] }} req -new -sha256 \
-newkey rsa:2048 -nodes -keyout ${:key} \
-subj "/O=${:organization}/OU=${:organizational_unit}" \
-reqexts SAN \
-config <(cat {{ software_parameter_dict['openssl-cnf'] }} \
<(printf "\n[SAN]\nsubjectAltName=IP:${:ip}")) \
-out ${:template-csr}'
fi
update-command = ${:command}
{#- Can be stopped on error, as does not rely on self provided service #}
stop-on-error = True
[directory]
recipe = slapos.cookbook:mkdirectory
bin = ${buildout:directory}/bin/
etc = ${buildout:directory}/etc/
srv = ${buildout:directory}/srv/
var = ${buildout:directory}/var/
tmp = ${buildout:directory}/tmp/
backup = ${:srv}/backup
log = ${:var}/log
run = ${:var}/run
service = ${:etc}/service
etc-run = ${:etc}/run
[slap_connection]
# Kept for backward compatibility
computer_id = ${slap-connection:computer-id}
partition_id = ${slap-connection:partition-id}
server_url = ${slap-connection:server-url}
software_release_url = ${slap-connection:software-release-url}
key_file = ${slap-connection:key-file}
cert_file = ${slap-connection:cert-file}
software/rapid-cdn/instance-rapid-cdn-frontend.cfg.jinja2
0 → 100644
View file @
bce9b4af
[buildout]
extends =
{{ software_parameter_dict['template-common'] }}
{{ software_parameter_dict['template-monitor2'] }}
{#- Manage frontends #}
{%- set FRONTEND_PARAMETER_KEY_LIST = ['url', 'domain', 'domain-icp-validation'] %} {#- matches schema-rapid-cdn-frontend-shared.input.json #}
{%- set BACKWARD_COMPATIBLE_FRONTEND_KEY_LIST = ['url'] %} {#- based on old JSON and also found important keys #}
{%- set FRONTEND_LIST = [] %}
{%- for slave in instance_parameter_dict['slave-instance-list'] %}
{%- set frontend = {} %}
{%- do frontend.__setitem__('title', slave['slave_title']) %}
{%- do frontend.__setitem__('reference', slave['slave_reference']) %}
{%- if '_' in slave %}
{%- do frontend.__setitem__('_', slave['_']) %}
{%- else %}
{%- for key in BACKWARD_COMPATIBLE_FRONTEND_KEY_LIST %}
{%- if key in slave %}
{%- do frontend.__setitem__(key, slave[key]) %}
{%- endif %}
{%- endfor %}
{%- endif %}
{%- do FRONTEND_LIST.append(frontend) %}
{%- endfor %}
{%- set PART_LIST = [] %}
{%- for frontend in FRONTEND_LIST | sort(attribute='title') %}
{%- if '_' in frontend %}
{%- set frontend_section_title = 'publish-frontend-' ~ hashlib.md5(frontend['title'].encode('utf-8')).hexdigest() %}
{%- set hostname = '%s.%s' % (frontend['reference'].replace("-", "").replace("_", "").lower(), slapparameter_dict['autogeneration-domain']) %}
{%- set rapid_url = 'https://%s/' % (hostname,) %}
{%- do PART_LIST.append(frontend_section_title) %}
[{{ frontend_section_title }}]
recipe = slapos.cookbook:publish.serialised
-slave-reference = {{ frontend['reference'] }}
rapid-url = {{ rapid_url }}
{%- endif %}
{%- endfor %}
{%- for frontend in FRONTEND_LIST | sort(attribute='title') %}
{%- if '_' not in frontend %}
{%- set frontend_section_title = 'publish-frontend-' ~ hashlib.md5(frontend['title'].encode('utf-8')).hexdigest() %}
{%- set hostname = '%s.%s' % (frontend['reference'].replace("-", "").replace("_", "").lower(), slapparameter_dict['autogeneration-domain']) %}
{%- set hostname = '%s.%s' % (frontend['reference'].replace("-", "").replace("_", "").lower(), slapparameter_dict['autogeneration-domain']) %}
{%- set url = 'http://%s/' % (hostname, ) %}
{%- set https_url = 'https://%s/' % (hostname,) %}
{%- do PART_LIST.append(frontend_section_title) %}
[{{ frontend_section_title }}]
recipe = slapos.cookbook:publish
-slave-reference = {{ frontend['reference'] }}
01-ATTENTION = Backward compatbility and obsolete frontend definition detected.
02-ATTENTION = The instance parameters are hidden in the UI, you see everything empty.
03-ATTENTION = Do not update parameters in the UI without checking the previous ones, as otherwise they will be lost.
04-ATTENTION = You can access your parameters by switching to XML view with "Show Parameter XML".
05-ATTENTION = Then you can wisely migrate them to the new way.
url = {{ url }}
secure_access = {{ https_url }}
site_url = {{ url }}
domain = {{ hostname }}
log-access-url = XXX-TODO
backend-client-caucase-url = XXX-TODO
key-generate-auth-url = XXX-TODO
replication_number = XXX-TODO
{%- endif %}
{%- endfor %}
{#- Request the tree #}
{%- macro sla(sla_key) %}
{%- set sla_key_length = sla_key | length %}
{%- for key in slapparameter_dict.keys() %}
{%- if key.startswith(sla_key) %}
sla-{{ key[sla_key_length:] }} = {{ slapparameter_dict[key] }}
{%- endif %}
{%- endfor %}
{%- endmacro %}
{%- for name in ['kedifa', 'human', 'log-aggregator'] %}
{%- set sla_key = '-sla-%s-computer_guid' % (name,) %}
{%- set software_type = 'rapid-cdn-' ~ name %}
{%- if not sla_key in slapparameter_dict %}
{%- do slapparameter_dict.__setitem__(sla_key, '${slap-connection:computer-id}') %}
{%- endif %}
{%- set software_key = '-%s-software-release-url' % (name,) %}
{%- if not software_key in slapparameter_dict %}
{%- do slapparameter_dict.__setitem__(software_key, '${slap-connection:software-release-url}') %}
{%- endif %}
[request-{{ name }}]
<= slap-connection
recipe = slapos.cookbook:requestoptional.serialised
name = {{ name }}
software-type = {{ software_type }}
software-url = {{ slapparameter_dict['-%s-software-release-url' % (name,)] }}
{{ sla('-sla-%s-' % (name,)) }}
{%- endfor %}
{%- set NODE_PART_LIST = [] %}
{%- for node, node_dict in slapparameter_dict.get('cluster-dict', {}).items() %}
{%- set node_name = 'node-' ~ node %}
{%- set part_name = 'request-' ~ node %}
{%- do NODE_PART_LIST.append(part_name) %}
{%- set sla_dict = node_dict.get('sla', {'computer_guid': '${slap-connection:computer-id}'}) %}
{%- set software_release_url = node_dict.pop('-software-release-url', '${slap-connection:software-release-url}') %}
[{{part_name }}]
<= slap-connection
recipe = slapos.cookbook:requestoptional.serialised
name = {{ node_name }}
software-type = rapid-cdn-pop
software-url = {{ software_release_url }}
config-node-name = {{ node_name }}
config-frontend-list = {{ dumps(FRONTEND_LIST) }}
config-kedifa-caucase-url = ${kedifa-caucase-promise:config-url}
{%- for sla_key, sla_value in sla_dict.items() %}
sla-{{ sla_key }} = {{ sla_value }}
{%- endfor %}
{%- endfor %}
[request-kedifa]
config-kedifa-port = {{ dumps(slapparameter_dict.get('-kedifa-port', 7890)) }}
config-caucase-port = {{ dumps(slapparameter_dict.get('-kedifa-caucase-port', 8090)) }}
config-frontend-list = {{ dumps(FRONTEND_LIST) }}
return =
caucase-url
master-key-generate-auth-url
master-key-upload-url
[kedifa-caucase-promise]
<= monitor-promise-base
module = check_url_available
name = ${:_buildout_section_name_}.py
config-http-code = 200
config-url = ${request-kedifa:connection-caucase-url}
[directory]
url-done = ${:var}/url-done
[url-ready]
recipe = slapos.recipe.build
depot = ${directory:url-done}
file = ${directory:url-done}/${:_buildout_section_name_}
marker = ${kedifa-config:marker}
init =
import os
if options['marker'] in options['url']:
if os.path.exists(options['file']):
os.unlink(options['file'])
else:
try:
with open(options['file'], 'w') as fh:
fh.write('')
except FileNotFoundError:
# directories are created during install, but promise runs during init
# so do nothing in such case
pass
[kedifa-master-key-generate-auth-url]
<= url-ready
url = ${request-kedifa:connection-master-key-generate-auth-url}
[kedifa-master-key-upload-url]
<= url-ready
url = ${request-kedifa:connection-master-key-upload-url}
[kedifa-master-key-promise]
<= monitor-promise-base
module = check_file_state
name = ${:_buildout_section_name_}.py
config-state = present
[kedifa-master-key-generate-auth-url-promise]
<= kedifa-master-key-promise
config-filename = ${kedifa-master-key-generate-auth-url:file}
url = ${kedifa-master-key-generate-auth-url:url}
[kedifa-master-key-upload-url-promise]
<= kedifa-master-key-promise
config-filename = ${kedifa-master-key-upload-url:file}
url = ${kedifa-master-key-upload-url:url}
[publish-information]
<= monitor-publish
recipe = slapos.cookbook:publish.serialised
xxx-replace-with-information-fetch-depends =
${request-human:name}
${request-log-aggregator:name}
{%- for node_part in NODE_PART_LIST %}
{{ '${' ~ node_part}}:name}
{%- endfor %}
kedifa-caucase-url = ${kedifa-caucase-promise:config-url}
key-generate-auth-url = ${kedifa-master-key-generate-auth-url-promise:url}
key-upload-url = ${kedifa-master-key-upload-url-promise:url}
[buildout]
parts +=
publish-information
{% for part in PART_LIST %}
{{ ' %s' % part }}
{% endfor %}
software/rapid-cdn/instance-rapid-cdn-human.cfg.jinja2
0 → 100644
View file @
bce9b4af
[buildout]
extends =
{{ software_parameter_dict['template-common'] }}
{{ software_parameter_dict['template-monitor2'] }}
software/rapid-cdn/instance-rapid-cdn-kedifa.cfg.jinja2
0 → 100644
View file @
bce9b4af
{%- import "caucase" as caucase with context %}
[buildout]
extends =
{{ software_parameter_dict['template-common'] }}
{{ software_parameter_dict['template-monitor2'] }}
parts +=
kedifa
caucased
caucase-updater
publish-information
[caucased]
hash-existing-files = ${buildout:directory}/software_release/buildout.cfg
{% set caucase_host = '[' ~ instance_parameter_dict['ipv6-random'] ~ ']' %}
{%- set caucase_netloc = caucase_host ~ ':' ~ slapparameter_dict['caucase-port'] -%}
{%- set caucase_url = 'http://' ~ caucase_netloc -%}
{{- caucase.caucased(
prefix='caucased',
buildout_bin_directory=software_parameter_dict['bin-directory'],
caucased_path='${directory:service}/caucased',
backup_dir='${directory:backup-caucased}',
data_dir='${directory:caucased}',
netloc=caucase_netloc,
tmp='${directory:tmp}',
service_auto_approve_count=0,
user_auto_approve_count=1,
key_len=2048
)}}
[caucase-updater]
hash-existing-files = ${buildout:directory}/software_release/buildout.cfg
{{ caucase.updater(
prefix='caucase-updater',
buildout_bin_directory=software_parameter_dict['bin-directory'],
updater_path='${directory:service}/caucase-updater',
url=caucase_url,
data_dir='${directory:srv}/caucase-updater',
crt_path='${kedifa-config:certificate}',
ca_path='${kedifa-config:ca-certificate}',
crl_path='${kedifa-config:crl}',
key_path='${kedifa-csr:key}',
on_renew='${kedifa-reloader:rendered}',
template_csr='${kedifa-csr:template-csr}'
)}}
[kedifa-csr]
<= csr
organizational_unit = Kedifa Partition
template-csr = ${kedifa-config:template-csr}
key = ${kedifa-config:key}
ip = ${kedifa-config:ip}
[kedifa-reloader]
recipe = slapos.recipe.template:jinja2
rendered = ${directory:etc-run}/kedifa-reloader
mode = 0700
template = inline:#!{{ software_parameter_dict['dash'] }}
{%- raw %}
{{ content }}
{%- endraw %}
command =
kill -HUP `cat ${kedifa-config:pidfile}`
context =
key content :command
[directory]
# KeDiFa directories
srv-kedifa = ${:srv}/kedifa
# CAUCASE directories
caucased = ${:srv}/caucased
backup-caucased = ${:backup}/caucased
# reservation
reservation = ${:srv}/reservation
[kedifa-config]
ip = {{ instance_parameter_dict['ipv6-random'] }}
port = {{ slapparameter_dict['kedifa-port'] }}
db = ${directory:srv-kedifa}/kedifa.sqlite
certificate = ${directory:srv-kedifa}/certificate.pem
key = ${:certificate}
ca-certificate = ${directory:srv-kedifa}/ca-certificate.pem
crl = ${directory:srv-kedifa}/crl.pem
template-csr = ${directory:srv-kedifa}/template-csr.pem
pidfile = ${directory:run}/kedifa.pid
logfile = ${directory:log}/kedifa.log
[kedifa]
recipe = slapos.cookbook:wrapper
command-line = {{ software_parameter_dict['kedifa'] }}
--ip ${kedifa-config:ip}
--port ${kedifa-config:port}
--db ${kedifa-config:db}
--certificate ${kedifa-config:certificate}
--ca-certificate ${kedifa-config:ca-certificate}
--crl ${kedifa-config:crl}
--pidfile ${kedifa-config:pidfile}
--logfile ${kedifa-config:logfile}
wrapper-path = ${directory:service}/kedifa
hash-existing-files = ${buildout:directory}/software_release/buildout.cfg
[master-auth-random-config]
file = ${directory:reservation}/${:_buildout_section_name_}
[master-auth-random-promise]
<= monitor-promise-base
module = check_file_state
name = ${:_buildout_section_name_}.py
config-filename = ${master-auth-random-config:file}
config-state = present
[master-auth-random]
recipe = slapos.recipe.build
kedifa-reserve = https://[${kedifa-config:ip}]:${kedifa-config:port}/reserve-id
cert = ${kedifa-config:certificate}
cacert = ${kedifa-config:ca-certificate}
result = ${kedifa-config:marker}
file = ${master-auth-random-promise:config-filename}
# The reservation has to happen during init, this slapos.recipe.build is used
# with python's requests instead of curl
init =
import errno
import os
import requests
try:
with open(options['file']) as fh:
options['result'] = fh.read()
except FileNotFoundError:
try:
result = requests.post(
options['kedifa-reserve'],
cert=options['cert'],
verify=options['cacert']
)
status_code = result.status_code
except Exception:
status_code = 500
if status_code == 201:
with open(options['file'], 'w') as fh:
fh.write(result.text)
options['result'] = result.text
[promise-kedifa-http-reply]
<= monitor-promise-base
module = check_url_available
name = kedifa-http-reply.py
# Kedifa replies 400 on /, so use it to be sure that Kedifa replied
config-http-code = 400
config-url = https://[${kedifa-config:ip}]:${kedifa-config:port}
config-ca-cert-file = ${kedifa-config:ca-certificate}
[publish-information]
recipe = slapos.cookbook:publish.serialised
caucase-url = {{ caucase_url }}
master-key-generate-auth-url = ${promise-kedifa-http-reply:config-url}/${master-auth-random:result}/generateauth
master-key-upload-url = ${promise-kedifa-http-reply:config-url}/${master-auth-random:result}?auth=
master-key-download-url = ${promise-kedifa-http-reply:config-url}/${master-auth-random:result}
software/rapid-cdn/instance-rapid-cdn-log-aggregator.cfg.jinja2
0 → 100644
View file @
bce9b4af
[buildout]
extends =
{{ software_parameter_dict['template-common'] }}
{{ software_parameter_dict['template-monitor2'] }}
software/rapid-cdn/instance-rapid-cdn-pop.cfg.jinja2
0 → 100644
View file @
bce9b4af
{%- import "caucase" as caucase with context %}
[buildout]
extends =
{{ software_parameter_dict['template-common'] }}
{{ software_parameter_dict['template-monitor2'] }}
parts +=
kedifa-caucase-updater
[directory]
kedifa-ssl = ${:srv}/kedifa-ssl
kedifa-caucase-updater = ${:srv}/kedifa-caucase-updater
[kedifa-config]
certificate = ${directory:kedifa-ssl}/certificate.pem
key = ${:certificate}
ca-certificate = ${directory:kedifa-ssl}/ca-certificate.pem
crl = ${directory:kedifa-ssl}/crl.pem
template-csr = ${directory:kedifa-ssl}/template-csr.pem
[kedifa-csr]
<= csr
organizational_unit = {{ slapparameter_dict['node-name'] }}
template-csr = ${kedifa-config:template-csr}
key = ${kedifa-config:key}
ip = {{ instance_parameter_dict['ipv6-random'] }}
[kedifa-caucase-updater]
hash-existing-files = ${buildout:directory}/software_release/buildout.cfg
{{ caucase.updater(
prefix='kedifa-caucase-updater',
buildout_bin_directory=software_parameter_dict['bin-directory'],
updater_path='${directory:service}/kedifa-caucase-updater',
url=slapparameter_dict['kedifa-caucase-url'],
data_dir='${directory:kedifa-caucase-updater}',
crt_path='${kedifa-config:certificate}',
ca_path='${kedifa-config:ca-certificate}',
crl_path='${kedifa-config:crl}',
key_path='${kedifa-csr:key}',
template_csr='${kedifa-csr:template-csr}'
)}}
software/rapid-cdn/instance.cfg.jinja2
0 → 100644
View file @
bce9b4af
[buildout]
extends = {{ software_parameter_dict['template-common'] }}
parts =
switch-softwaretype
[switch-softwaretype]
recipe = slapos.cookbook:switch-softwaretype
{#- Cluster itself #}
rapid-cdn-frontend = template-rapid-cdn-frontend:rendered
rapid-cdn-pop = template-rapid-cdn-pop:rendered
rapid-cdn-human = template-rapid-cdn-human:rendered
rapid-cdn-kedifa = template-rapid-cdn-kedifa:rendered
rapid-cdn-log-aggregator = template-rapid-cdn-log-aggregator:rendered
{#- Defaults... #}
default = ${:rapid-cdn-frontend}
RootSoftwareInstance = ${:rapid-cdn-frontend}
[jinja2-template-base]
recipe = slapos.recipe.template:jinja2
rendered = ${buildout:directory}/${:filename}
extensions = jinja2.ext.do
extra-context =
context =
import json json
import hashlib hashlib
key slapparameter_dict slap-configuration:configuration
section instance_parameter_dict slap-configuration
section software_parameter_dict software-parameter-section
${:extra-context}
import-list =
file caucase software-parameter-section:caucase-jinja2-library
[template-base]
<= jinja2-template-base
filename = ${:_buildout_section_name_}.cfg
{%- for template in software_parameter_dict.keys() %}
{%- if template.startswith('template-rapid-cdn-') %}
[{{ template }}]
<= template-base
template = {{ software_parameter_dict[template] }}
{%- endif %}
{%- endfor %}
[software-parameter-section]
{%- for key, value in software_parameter_dict.items() %}
{{ key }} = {{ dumps(value) }}
{% endfor -%}
[slap-configuration]
# Fetches parameters defined in SlapOS Master for this instance.
# Always the same.
recipe = slapos.cookbook:slapconfiguration.serialised
computer = ${slap-connection:computer-id}
partition = ${slap-connection:partition-id}
url = ${slap-connection:server-url}
key = ${slap-connection:key-file}
cert = ${slap-connection:cert-file}
software/rapid-cdn/software.cfg
View file @
bce9b4af
[buildout]
extends =
buildout.hash.cfg
../../stack/caucase/buildout.cfg
../../stack/slapos.cfg
../../component/python3/buildout.cfg
parts +=
template
[python]
part = python3
[template-render-base]
recipe = slapos.recipe.template:jinja2
template = ${:_profile_base_location_}/${:filename}
rendered = ${buildout:directory}/${:_buildout_section_name_}.cfg
mode = 0644
context =
section software_parameter_dict software-parameter-section
${:extra-context}
extra-context =
[template]
<= template-render-base
[template-common]
<= template-render-base
[template-download-base]
recipe = slapos.recipe.build:download
url = ${:_profile_base_location_}/${:filename}
mode = 0644
[template-rapid-cdn-frontend]
<= template-download-base
[template-rapid-cdn-human]
<= template-download-base
[template-rapid-cdn-pop]
<= template-download-base
[template-rapid-cdn-kedifa]
<= template-download-base
[template-rapid-cdn-log-aggregator]
<= template-download-base
# kedifa unreleased version
[kedifa-repository]
recipe = slapos.recipe.build:gitclone
repository = https://lab.nexedi.com/luke/kedifa.git
git-executable = ${git:location}/bin/git
revision = ea8c6479c4c74bf87509bb7de847f43301736428
[kedifa-develop]
recipe = zc.recipe.egg:develop
setup = ${kedifa-repository:location}
egg = kedifa
[kedifa]
recipe = zc.recipe.egg
eggs =
${python-cryptography:egg}
${kedifa-develop:egg}
[software-parameter-section]
# This section passes binaries and other software provided outputs to all
# profiles. In the same time allows to gather all parameters in one place.
# Buildout parameters
develop-eggs-directory = ${buildout:develop-eggs-directory}
eggs-directory = ${buildout:eggs-directory}
# Software
bin-directory = ${buildout:bin-directory}
caucase = ${caucase-eggs:bin-directory}/caucase
caucase-probe = ${caucase-eggs:bin-directory}/caucase-probe
caucase-updater = ${caucase-eggs:bin-directory}/caucase-updater
caucased = ${caucase-eggs:bin-directory}/caucased
dash = ${dash-output:dash}
kedifa = ${kedifa:bin-directory}/kedifa
kedifa-updater = ${kedifa:bin-directory}/kedifa-updater
openssl = ${openssl:location}/bin/openssl
openssl-cnf = ${openssl:location}/etc/ssl/openssl.cnf
# Libraries
caucase-jinja2-library = ${caucase-jinja2-library:target}
# Templates
template-common = ${template-common:rendered}
template-monitor2 = ${monitor2-template:rendered}
template-rapid-cdn-frontend = ${template-rapid-cdn-frontend:target}
template-rapid-cdn-human = ${template-rapid-cdn-human:target}
template-rapid-cdn-pop = ${template-rapid-cdn-pop:target}
template-rapid-cdn-kedifa = ${template-rapid-cdn-kedifa:target}
template-rapid-cdn-log-aggregator = ${template-rapid-cdn-log-aggregator:target}
[versions]
#kedifa = kedifa-unreleased-version
# modernish zc.lockfile is required by kedifa
zc.lockfile = 1.4
software/rapid-cdn/test/setup.py
View file @
bce9b4af
...
...
@@ -43,6 +43,8 @@ setup(
install_requires
=
[
'slapos.core'
,
'slapos.libnetworkcache'
,
'requests'
,
'caucase'
,
],
zip_safe
=
True
,
test_suite
=
'test'
,
...
...
software/rapid-cdn/test/test.py
View file @
bce9b4af
...
...
@@ -25,7 +25,12 @@
#
##############################################################################
import
json
import
os
import
shutil
import
subprocess
import
tempfile
import
slapos.slap.standalone
from
slapos.testing.testcase
import
makeModuleSetUpAndTestCaseClass
...
...
@@ -34,6 +39,109 @@ setUpModule, SlapOSInstanceTestCase = makeModuleSetUpAndTestCaseClass(
os
.
path
.
join
(
os
.
path
.
dirname
(
__file__
),
'..'
,
'software.cfg'
)))
class
CaucaseBinaryClient
(
object
):
# Class to mimic user, which is using caucase binaries and some directories
# to be a client, thus real binaries are used with executable call examples
def
__init__
(
self
,
path
,
url
):
self
.
path
=
path
os
.
mkdir
(
self
.
path
)
self
.
url
=
url
def
makepath
(
p
):
return
os
.
path
.
join
(
self
.
path
,
p
)
self
.
client_csr
=
makepath
(
'client.csr.pem'
)
self
.
client
=
makepath
(
'client.pem'
)
self
.
ca_crt
=
makepath
(
'ca-crt.pem'
)
self
.
user_ca_crt
=
makepath
(
'user-ca-crt.pem'
)
self
.
crl
=
makepath
(
'crl.pem'
)
self
.
user_crl
=
makepath
(
'user-crl.pem'
)
subprocess
.
check_call
([
"openssl"
,
"req"
,
"-out"
,
self
.
client_csr
,
"-new"
,
"-newkey"
,
"rsa:2048"
,
"-nodes"
,
"-keyout"
,
self
.
client
,
"-subj"
,
"/CN=user"
])
self
.
cau_list
=
[
"caucase"
,
"--ca-url"
,
self
.
url
,
"--ca-crt"
,
self
.
ca_crt
,
"--user-ca-crt"
,
self
.
user_ca_crt
,
"--crl"
,
self
.
crl
,
"--user-crl"
,
self
.
user_crl
]
self
.
user_cau_list
=
self
.
cau_list
+
[
"--mode"
,
"user"
]
self
.
service_cau_list
=
self
.
cau_list
+
[
"--user-key"
,
self
.
client
,
"--mode"
,
"service"
]
output
=
subprocess
.
check_output
(
self
.
user_cau_list
+
[
"--send-csr"
,
self
.
client_csr
])
subprocess
.
check_call
(
self
.
user_cau_list
+
[
"--get-crt"
,
output
.
split
()[
0
],
self
.
client
])
def
signServiceCsr
(
self
,
ou
):
for
line
in
subprocess
.
check_output
(
self
.
service_cau_list
+
[
"--list-csr"
]).
splitlines
():
splitted
=
line
.
decode
().
split
(
'|'
)
if
len
(
splitted
)
==
2
:
if
'OU=%s)'
%
(
ou
,)
in
splitted
[
1
]:
csr_id
=
splitted
[
0
].
strip
()
subprocess
.
check_call
(
self
.
service_cau_list
+
[
"--sign-csr"
,
csr_id
])
break
class
Test
(
SlapOSInstanceTestCase
):
def
test
(
self
):
self
.
fail
(
'TODO'
)
# full diffs are very informative for those tests
maxDiff
=
None
# as instantiation is split in chunks with cluster administrator operations,
# let it run a bit rarely
instance_max_retry
=
5
@
classmethod
def
_setUpClass
(
cls
):
cls
.
work_dir
=
tempfile
.
mkdtemp
()
try
:
super
().
_setUpClass
()
except
slapos
.
slap
.
standalone
.
SlapOSNodeInstanceError
:
pass
# Cluster is initially setup, now it's required to allow joining nodes,
# which is usually cluster administrator responsibility
connection_parameter_dict
=
json
.
loads
(
cls
.
requestDefaultInstance
().
getConnectionParameterDict
()[
'_'
])
cls
.
kedifa_caucase_client
=
CaucaseBinaryClient
(
os
.
path
.
join
(
cls
.
work_dir
,
'kedifa_caucase'
),
connection_parameter_dict
[
'kedifa-caucase-url'
])
cls
.
kedifa_caucase_client
.
signServiceCsr
(
"Kedifa Partition"
)
# Time travel to the future: restart all caucase updaters, so they will
# pick up just signed service CSR; note that in reality the cluster
# administrator can wait for the caucase updater to kick in
with
cls
.
slap
.
instance_supervisor_rpc
as
instance_supervisor_rpc
:
for
process
in
instance_supervisor_rpc
.
getAllProcessInfo
():
if
'caucase-updater'
in
process
[
'name'
]:
process_id
=
'%(group)s:%(name)s'
%
process
instance_supervisor_rpc
.
stopProcess
(
process_id
)
instance_supervisor_rpc
.
startProcess
(
process_id
)
cls
.
waitForInstance
()
@
classmethod
def
tearDownClass
(
cls
):
super
().
tearDownClass
()
shutil
.
rmtree
(
cls
.
work_dir
)
def
assertMonitorSetupUrl
(
self
,
value
):
# TODO: check that monitor can correctly acces whole cluster
pass
def
test_connection_parameter
(
self
):
connection_parameter_dict
=
self
.
requestDefaultInstance
(
).
getConnectionParameterDict
()
self
.
assertIn
(
'_'
,
connection_parameter_dict
)
json_parmeter_dict
=
json
.
loads
(
connection_parameter_dict
.
pop
(
'_'
))
self
.
assertEqual
({},
connection_parameter_dict
)
self
.
assertNotIn
(
'__NotReadyYet__'
,
json_parmeter_dict
.
pop
(
'key-generate-auth-url'
))
self
.
assertNotIn
(
'__NotReadyYet__'
,
json_parmeter_dict
.
pop
(
'key-upload-url'
))
self
.
assertMonitorSetupUrl
(
json_parmeter_dict
.
pop
(
'monitor-setup-url'
))
self
.
assertEqual
(
{
'kedifa-caucase-url'
:
'http://[::2]:8090'
,
'monitor-base-url'
:
'https://[::2]:8196'
,
'xxx-replace-with-information-fetch-depends'
:
'human
\
n
log-aggregator'
},
json_parmeter_dict
)
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment