Skip to content

slapos
slapos
Commit be08f156 authored by Alain Takoudjou's avatar Alain Takoudjou
Browse files
Options

Update Release Candidate

parents 121aff64 06d4db50
Expand all Hide whitespace changes
Inline Side-by-side
Showing with 1119 additions and 194 deletions
software/caddy-frontend/TODO.rst
View file @ be08f156
Generally things to be done with ``caddy-frontend``:
* tests: add assertion with results of promises in etc/promise for each partition
* generated files: ``| trim`` values (like ``slave_password[slave]`` in ``templates/template-log-access.conf.in``) in generated configuration files to have them renfered correctly
* check the whole frontend slave snippet with ``caddy -validate`` during buildout run, and reject if does not pass validation
* check that all options from ``instance-slave-caddy-input-schema.json`` are safe to be used
* ``apache-ca-certificate`` shall be merged with ``apache-certificate``
* ``apache-ca-certificate`` shall be appended to ``apache-certificate`` if not already there
......@@ -26,7 +26,6 @@ Generally things to be done with ``caddy-frontend``:
* ``apache-ca-certificate``
* ``apache-certificate`` and ``apache-key``
* change ``switch-softwaretype`` to way how ``software/erp5`` does, which will help with dropping jinja2 template for ``caddy-wrapper``, which is workaround for current situation, cf `note_62678 <https://lab.nexedi.com/nexedi/slapos/merge_requests/312#note_62678>`_
* use `slapos!326 <https://lab.nexedi.com/nexedi/slapos/merge_requests/326>`_, and especially `note about complex restart scenarios <https://lab.nexedi.com/nexedi/slapos/merge_requests/326#note_60198>`_, instead of self-developed graceful restart scripts
* move out ``test/utils.py`` and use it from shared python distribution
* provide various tricks for older browsers::
......@@ -61,7 +60,7 @@ Generally things to be done with ``caddy-frontend``:
<FilesMatch "\.(cgi|shtml|phtml|php)$">
SSLOptions +StdEnvVars
</FilesMatch>
* reduce the time of configuration validation (in ``instance-apache-frontend.cfg`` sections ``[configtest]``, ``[caddy-configuration]``, ``[nginx-configuration]``), as it is not scalable on frontend with 2000+ slaves (takes few minutes instead of few, < 5, seconds), issue posted `upstream <https://github.com/mholt/caddy/issues/2220>`_
* reduce the time of configuration validation (in ``instance-apache-frontend.cfg.in`` sections ``[configtest]``, ``[caddy-configuration]``, ``[nginx-configuration]``), as it is not scalable on frontend with 2000+ slaves (takes few minutes instead of few, < 5, seconds), issue posted `upstream <https://github.com/mholt/caddy/issues/2220>`_
* drop ``6tunnel`` and use ``bind`` in Caddy configuration, as soon as multiple binds will be possible, tracked in upstream `bind: support multiple values <https://github.com/mholt/caddy/pull/2128>`_ and `ipv6: does not bind on ipv4 and ipv6 for sites that resolve to both <https://github.com/mholt/caddy/issues/864>`_
* use caddy-frontend in `standalone style playbooks <https://lab.nexedi.com/nexedi/slapos.package/tree/master/playbook/roles/standalone-shared>`_
* ensure `QUIC <https://en.wikipedia.org/wiki/QUIC>`_ is used by caddy
......
software/caddy-frontend/buildout.hash.cfg
View file @ be08f156
......@@ -13,20 +13,24 @@
# section inheritance (< = ...) are NOT supported (but you should really
# not need these here).
[template]
filename = instance.cfg
md5sum = b73505ae80d6325a244f5094f8edc0ae
filename = instance.cfg.in
md5sum = d649e128d36cf76f870c189c53985569
[template-common]
filename = instance-common.cfg.in
md5sum = c801b7f9f11f0965677c22e6bbe9281b
[template-apache-frontend]
filename = instance-apache-frontend.cfg
md5sum = b170d0987563b481eb71cf705c3658ab
filename = instance-apache-frontend.cfg.in
md5sum = 64fb8005a62f0a3a9987de2e336b68e1
[template-apache-replicate]
filename = instance-apache-replicate.cfg.in
md5sum = 27e98547061bd81e5f84cb7dd21b683b
md5sum = 8d34141a9cd1e51462aba845c7bea85b
[template-slave-list]
filename = templates/apache-custom-slave-list.cfg.in
md5sum = fb6c93f42f232e381174a5951c3fc222
md5sum = 8f29aaf247a6b8354292c78abe7a5ad6
[template-slave-configuration]
filename = templates/custom-virtualhost.conf.in
......@@ -34,15 +38,15 @@ md5sum = 54ae95597a126ae552c3a913ddf29e5e
[template-replicate-publish-slave-information]
filename = templates/replicate-publish-slave-information.cfg.in
md5sum = 8d318af17da5631d4242c0d6d1531066
md5sum = 6a308c29b54d53cfd82ae23ba77a35dd
[template-caddy-frontend-configuration]
filename = templates/Caddyfile.in
md5sum = 6689d96fc18d9aad78d77fe87770d4da
md5sum = 7c987ad75fcce6f5b925c7696ff41971
[template-custom-slave-list]
filename = templates/apache-custom-slave-list.cfg.in
md5sum = fb6c93f42f232e381174a5951c3fc222
md5sum = 8f29aaf247a6b8354292c78abe7a5ad6
[caddy-backend-url-validator]
filename = templates/caddy-backend-url-validator.in
......@@ -54,15 +58,15 @@ md5sum = f20d6c3d2d94fb685f8d26dfca1e822b
[template-default-slave-virtualhost]
filename = templates/default-virtualhost.conf.in
md5sum = 6da56d875f5cf396f8fd0685cf1a9a7a
md5sum = 8ed87061b9e20e2ad74aae9f80d1b53d
[template-cached-slave-virtualhost]
filename = templates/cached-virtualhost.conf.in
md5sum = 0e7d8df879ec363f771740d017cb7512
md5sum = f149ab15334d7d15d8c525f02fc4d968
[template-log-access]
filename = templates/template-log-access.conf.in
md5sum = cbf492b2fd8a955b0f92eb512fe9163f
md5sum = f2a74f88c7248f199011fa9ec6182f73
[template-empty]
filename = templates/empty.in
......@@ -72,10 +76,6 @@ md5sum = c2314c3a9c3412a38d14b312d3df83c1
filename = templates/wrapper.in
md5sum = 8cde04bfd0c0e9bd56744b988275cfd8
[template-caddy-wrapper]
filename = templates/caddy-wrapper.in
md5sum = c5816275757124613920078b6bec1caf
[template-trafficserver-records-config]
filename = templates/trafficserver/records.config.jinja2
md5sum = 84baef0a49c9a65e8f2d2ffdb8c1d39c
......@@ -90,11 +90,11 @@ md5sum = fadb2fcaf0f2b4fe735617fac222f7ed
[template-nginx-eventsource-slave-virtualhost]
filename = templates/nginx-eventsource-slave.conf.in
md5sum = 69d65e461cd7cd5ef5b1ccd0098b50c8
md5sum = 176cbca2070734a185a7ae5a4d1181c5
[template-nginx-notebook-slave-virtualhost]
filename = templates/nginx-notebook-slave.conf.in
md5sum = 21a102ac2ee98f9a7f168fa0a1390068
md5sum = e018935e2cec2368991f743cab725741
[template-apache-lazy-script-call]
filename = templates/apache-lazy-script-call.sh.in
......
software/caddy-frontend/common.cfg
View file @ be08f156
......@@ -34,16 +34,71 @@ eggs +=
websockify
erp5.util
[template-common]
recipe = slapos.recipe.template:jinja2
template = ${:_profile_base_location_}/instance-common.cfg.in
rendered = ${buildout:directory}/instance-common.cfg
mode = 0644
context =
key develop_eggs_directory buildout:develop-eggs-directory
key eggs_directory buildout:eggs-directory
[template-frontend-parameter-section]
common_profile = ${template-common:rendered}
bin_directory = ${buildout:bin-directory}
sixtunnel = ${6tunnel:location}
caddy = ${caddy:output}
caddy_location = ${caddy:location}
curl = ${curl:location}
dash = ${dash:location}
dcron = ${dcron:location}
gzip = ${gzip:location}
logrotate = ${logrotate:location}
openssl = ${openssl:location}
trafficserver = ${trafficserver:location}
monitor_template = ${monitor-template:output}
template_cached_slave_virtualhost = ${template-cached-slave-virtualhost:target}
template_caddy_frontend_configuration = ${template-caddy-frontend-configuration:target}
template_caddy_graceful_script = ${template-caddy-graceful-script:target}
template_caddy_lazy_script_call = ${template-caddy-lazy-script-call:target}
template_default_slave_virtualhost = ${template-default-slave-virtualhost:target}
template_empty = ${template-empty:target}
template_log_access = ${template-log-access:target}
template_nging_configuration = ${template-nginx-configuration:output}
template_nginx_eventsource_slave_virtualhost = ${template-nginx-eventsource-slave-virtualhost:target}
template_nginx_notebook_slave_virtualhost = ${template-nginx-notebook-slave-virtualhost:target}
template_not_found_html = ${template-not-found-html:target}
template_slave_configuration = ${template-slave-configuration:target}
template_slave_list = ${template-slave-list:target}
template_trafficserver_records_config = ${template-trafficserver-records-config:location}
template_trafficserver_records_config_filename = ${template-trafficserver-records-config:filename}
template_trafficserver_records_config_location = ${template-trafficserver-records-config:location}
template_trafficserver_storage_config_filename = ${template-trafficserver-storage-config:filename}
template_trafficserver_storage_config_location = ${template-trafficserver-storage-config:location}
template_wrapper = ${template-wrapper:output}
[template]
recipe = slapos.recipe.template
url = ${:_profile_base_location_}/instance.cfg
output = ${buildout:directory}/template.cfg
recipe = slapos.recipe.template:jinja2
template = ${:_profile_base_location_}/instance.cfg.in
rendered = ${buildout:directory}/template.cfg
mode = 0644
context =
key common_profile template-common:rendered
key monitor2_template monitor2-template:rendered
key template_caddy_frontend template-caddy-frontend:target
key template_caddy_replicate template-caddy-replicate:target
key template_replicate_publish_slave_information template-replicate-publish-slave-information:target
key caddy_backend_url_validator caddy-backend-url-validator:output
section template_frontend_parameter_dict template-frontend-parameter-section
[template-caddy-frontend]
recipe = slapos.recipe.template
url = ${:_profile_base_location_}/instance-apache-frontend.cfg
output = ${buildout:directory}/template-caddy-frontend.cfg
recipe = slapos.recipe.build:download
url = ${:_profile_base_location_}/instance-apache-frontend.cfg.in
mode = 0644
[caddy-backend-url-validator]
......@@ -103,12 +158,6 @@ filename = template-log-access.conf.in
<=download-template
filename = empty.in
[template-caddy-wrapper]
recipe = slapos.recipe.template
url = ${:_profile_base_location_}/templates/caddy-wrapper.in
output = ${buildout:directory}/template-caddy-wrapper.cfg
mode = 0644
[template-wrapper]
recipe = slapos.recipe.template
url = ${:_profile_base_location_}/templates/wrapper.in
......@@ -116,22 +165,14 @@ output = ${buildout:directory}/template-wrapper.cfg
mode = 0644
[template-trafficserver-records-config]
recipe = hexagonit.recipe.download
ignore-existing = true
<=download-template
url = ${:_profile_base_location_}/templates/trafficserver/${:filename}
location = ${buildout:parts-directory}/${:_buildout_section_name_}
filename = records.config.jinja2
download-only = true
mode = 0644
[template-trafficserver-storage-config]
recipe = hexagonit.recipe.download
ignore-existing = true
<=download-template
url = ${:_profile_base_location_}/templates/trafficserver/${:filename}
location = ${buildout:parts-directory}/${:_buildout_section_name_}
filename = storage.config.jinja2
download-only = true
mode = 0644
# NGINX Configuration
[template-nginx-configuration]
......
software/caddy-frontend/instance-apache-replicate.cfg.in
View file @ be08f156
......@@ -6,10 +6,7 @@ rendered = ${buildout:directory}/${:filename}
extra-context =
context =
import json_module json
key eggs_directory buildout:eggs-directory
key develop_eggs_directory buildout:develop-eggs-directory
key slap_software_type slap-parameter:slap_software_type
key slave_instance_list slap-parameter:slave_instance_list
raw common_profile {{ common_profile }}
${:extra-context}
{% set part_list = [] %}
......@@ -173,7 +170,9 @@ monitor-url-list +=
{% endfor %}
[buildout]
extends = {{ template_monitor }}
extends =
{{ common_profile }}
{{ template_monitor }}
parts =
monitor-base
publish-slave-information
......@@ -182,23 +181,4 @@ parts =
{{ ' %s' % part }}
{% endfor %}
# publish-information
eggs-directory = {{ eggs_directory }}
develop-eggs-directory = {{ develop_eggs_directory }}
offline = true
[slap_connection]
# Kept for backward compatibility
computer_id = ${slap-connection:computer-id}
partition_id = ${slap-connection:partition-id}
server_url = ${slap-connection:server-url}
software_release_url = ${slap-connection:software-release-url}
key_file = ${slap-connection:key-file}
cert_file = ${slap-connection:cert-file}
[slap-parameter]
slave_instance_list =
-frontend-quantity = 1
-frontend-type = single-default
{% endif %}
software/caddy-frontend/instance-common.cfg.in 0 → 100644
View file @ be08f156
[buildout]
eggs-directory = {{ eggs_directory }}
develop-eggs-directory = {{ develop_eggs_directory }}
offline = true
[slap_connection]
# Kept for backward compatibility
computer_id = ${slap-connection:computer-id}
partition_id = ${slap-connection:partition-id}
server_url = ${slap-connection:server-url}
software_release_url = ${slap-connection:software-release-url}
key_file = ${slap-connection:key-file}
cert_file = ${slap-connection:cert-file}
\ No newline at end of file
software/caddy-frontend/instance.cfg deleted 100644 → 0
View file @ 121aff64
[buildout]
parts =
dynamic-template-caddy-replicate
switch-softwaretype
eggs-directory = ${buildout:eggs-directory}
develop-eggs-directory = ${buildout:develop-eggs-directory}
offline = true
[slap-parameters]
recipe = slapos.cookbook:slapconfiguration
computer = $${slap-connection:computer-id}
partition = $${slap-connection:partition-id}
url = $${slap-connection:server-url}
key = $${slap-connection:key-file}
cert = $${slap-connection:cert-file}
[jinja2-template-base]
recipe = slapos.recipe.template:jinja2
rendered = $${buildout:directory}/$${:filename}
extra-context =
context =
import json_module json
key eggs_directory buildout:eggs-directory
key develop_eggs_directory buildout:develop-eggs-directory
key slap_software_type slap-parameters:slap-software-type
key slapparameter_dict slap-parameters:configuration
key slave_instance_list slap-parameters:slave-instance-list
$${:extra-context}
[switch-softwaretype]
recipe = slapos.cookbook:softwaretype
default = $${dynamic-template-caddy-replicate:rendered}
RootSoftwareInstance = $${dynamic-template-caddy-replicate:rendered}
custom-personal = $${dynamic-template-caddy-replicate:rendered}
single-default = ${template-caddy-frontend:output}
single-custom-personal = ${template-caddy-frontend:output}
replicate = $${dynamic-template-caddy-replicate:rendered}
[dynamic-template-caddy-replicate]
< = jinja2-template-base
template = ${template-caddy-replicate:target}
filename = instance-caddy-replicate.cfg
extensions = jinja2.ext.do
extra-context =
import subprocess_module subprocess
raw caddy_backend_url_validator ${caddy-backend-url-validator:output}
raw template_publish_slave_information ${template-replicate-publish-slave-information:target}
# Must match the key id in [switch-softwaretype] which uses this section.
raw software_type RootSoftwareInstance-default-custom-personal-replicate
raw template_monitor ${monitor2-template:rendered}
software/caddy-frontend/instance.cfg.in 0 → 100644
View file @ be08f156
[buildout]
extends = {{ common_profile }}
parts =
dynamic-template-caddy-replicate
switch-softwaretype
[jinja2-template-base]
recipe = slapos.recipe.template:jinja2
rendered = ${buildout:directory}/${:filename}
extra-context =
context =
import json_module json
key slap_software_type instance-parameter:slap-software-type
key slapparameter_dict instance-parameter:configuration
key slave_instance_list instance-parameter:slave-instance-list
section instance_parameter instance-parameter
${:extra-context}
[switch-softwaretype]
recipe = slapos.cookbook:softwaretype
default = ${dynamic-template-caddy-replicate:rendered}
RootSoftwareInstance = ${dynamic-template-caddy-replicate:rendered}
custom-personal = ${dynamic-template-caddy-replicate:rendered}
single-default = ${dynamic-template-caddy-frontend:rendered}
single-custom-personal = ${dynamic-template-caddy-frontend:rendered}
replicate = ${dynamic-template-caddy-replicate:rendered}
[dynamic-template-caddy-frontend-parameters]
{% for key,value in template_frontend_parameter_dict.iteritems() %}
{{ key }} = {{ dumps(value) }}
{% endfor -%}
[dynamic-template-caddy-frontend]
< = jinja2-template-base
template = {{ template_caddy_frontend }}
filename = instance-caddy-frontend.cfg
extensions = jinja2.ext.do
extra-context =
section parameter_dict dynamic-template-caddy-frontend-parameters
[dynamic-template-caddy-replicate]
< = jinja2-template-base
template = {{ template_caddy_replicate }}
filename = instance-caddy-replicate.cfg
extensions = jinja2.ext.do
extra-context =
import subprocess_module subprocess
raw caddy_backend_url_validator {{ caddy_backend_url_validator }}
raw template_publish_slave_information {{ template_replicate_publish_slave_information }}
# Must match the key id in [switch-softwaretype] which uses this section.
raw software_type RootSoftwareInstance-default-custom-personal-replicate
raw template_monitor {{ monitor2_template }}
raw common_profile {{ common_profile }}
[instance-parameter]
# Fetches parameters defined in SlapOS Master for this instance.
# Always the same.
recipe = slapos.cookbook:slapconfiguration.serialised
computer = ${slap-connection:computer-id}
partition = ${slap-connection:partition-id}
url = ${slap-connection:server-url}
key = ${slap-connection:key-file}
cert = ${slap-connection:cert-file}
# Define default parameter(s) that will be used later, in case user didn't
# specify it
# All parameters are available through the configuration.XX syntax.
# All possible parameters should have a default.
configuration.domain = example.org
configuration.public-ipv4 =
configuration.port = 4443
configuration.plain_http_port = 8080
configuration.plain_nginx_port = 8081
configuration.nginx_port = 9443
# BBB: apache_custom_https and apache_custom_http
configuration.apache_custom_https = ""
configuration.apache_custom_http = ""
configuration.caddy_custom_https = ""
configuration.caddy_custom_http = ""
configuration.apache-key =
configuration.apache-certificate =
configuration.apache-ca-certificate =
configuration.open-port = 80 443
configuration.extra_slave_instance_list =
configuration.disk-cache-size = 8G
configuration.ram-cache-size = 1G
configuration.trafficserver-autoconf-port = 8083
configuration.trafficserver-mgmt-port = 8084
configuration.re6st-verification-url = http://[2001:67c:1254:4::1]/index.html
configuration.enable-http2-by-default = true
configuration.mpm-graceful-shutdown-timeout = 5
configuration.monitor-httpd-port = 8072
configuration.frontend-name =
\ No newline at end of file
software/caddy-frontend/templates/Caddyfile.in
View file @ be08f156
......@@ -34,7 +34,7 @@ https://[{{ global_ipv6 }}]:{{ https_port }}/server-status, https://{{ local_ipv
# Compress the output
gzip
bind {{ local_ipv4 }}
basicauth "{{ username }}" {{ password }} {
basicauth "{{ username }}" {{ password | trim }} {
"Server Status"
/
}
......
software/caddy-frontend/templates/apache-custom-slave-list.cfg.in
View file @ be08f156
......@@ -20,8 +20,7 @@ recipe = slapos.recipe.template:jinja2
extensions = jinja2.ext.do
extra-context =
context =
key eggs_directory buildout:eggs-directory
key develop_eggs_directory buildout:develop-eggs-directory
raw common_profile {{ common_profile }}
${:extra-context}
{% do logrotate_dict.pop('recipe') %}
......@@ -194,6 +193,13 @@ cert-content = {{ dumps(slave_instance.get('ssl_crt')) }}
{# ########################################## #}
{# Set Slave Configuration #}
[{{ slave_configuration_section_name }}]
https_port = {{ dumps(https_port) }}
http_port = {{ dumps(http_port) }}
local_ipv4 = {{ dumps(local_ipv4) }}
nginx_http_port = {{ dumps(nginx_http_port) }}
nginx_https_port = {{ dumps(nginx_https_port) }}
cached_port = {{ dumps(cached_port) }}
ssl_cached_port = {{ (ssl_cached_port) }}
{# BBB: apache_custom_https and apache_custom_http #}
{% set caddy_custom_http = ((slave_instance.pop('caddy_custom_http', slave_instance.pop('apache_custom_http', ''))) % slave_parameter_dict) %}
{% set caddy_custom_https = ((slave_instance.pop('caddy_custom_https', slave_instance.pop('apache_custom_https', ''))) % slave_parameter_dict) %}
......@@ -225,11 +231,6 @@ template = {{ template_default_slave_configuration }}
filename = {{ '%s.conf' % slave_reference }}
extra-context =
raw https_port {{ https_port }}
raw http_port {{ http_port }}
raw local_ipv4 {{ local_ipv4 }}
raw nginx_http_port {{ nginx_http_port }}
raw nginx_https_port {{ nginx_https_port }}
section slave_parameter {{ slave_configuration_section_name }}
{{ '\n' }}
......@@ -316,9 +317,6 @@ rendered = {{ caddy_cached_configuration_directory }}/${:filename}
extensions = jinja2.ext.do
extra-context =
section slave_parameter {{ slave_configuration_section_name }}
raw cached_port {{ cached_port }}
raw ssl_cached_port {{ ssl_cached_port }}
raw local_ipv4 {{ local_ipv4 }}
{{ '\n' }}
{% endfor %}
......@@ -365,6 +363,19 @@ ipv4-port = {{ nginx_https_port }}
ipv6-port = {{ nginx_https_port }}
{# Define log access #}
[caddy-log-access-parameters]
caddy_log_directory = {{ dumps(caddy_log_directory) }}
caddy_configuration_directory = {{ dumps(caddy_configuration_directory) }}
local_ipv4 = {{ dumps(local_ipv4) }}
global_ipv6 = {{ dumps(global_ipv6) }}
https_port = {{ dumps(https_port) }}
http_port = {{ dumps(http_port) }}
login_certificate = {{ dumps(login_certificate) }}
login_key = {{ dumps(login_key) }}
access_log = {{ dumps(access_log) }}
error_log = {{ dumps(error_log) }}
not_found_file = {{ dumps(not_found_file) }}
[caddy-log-access]
< = jinja2-template-base
template = {{frontend_configuration.get('template-log-access')}}
......@@ -372,17 +383,7 @@ rendered = {{frontend_configuration.get('log-access-configuration')}}
extra-context =
section slave_log_directory slave-log-directory-dict
section slave_password slave-password
raw caddy_log_directory {{caddy_log_directory}}
raw caddy_configuration_directory {{caddy_configuration_directory}}
raw local_ipv4 {{ local_ipv4 }}
raw global_ipv6 {{ global_ipv6 }}
raw https_port {{ https_port }}
raw http_port {{ http_port }}
raw login_certificate {{ login_certificate }}
raw login_key {{ login_key }}
raw access_log {{ access_log }}
raw error_log {{ error_log }}
raw not_found_file {{ not_found_file }}
section parameter_dict caddy-log-access-parameters
{# Publish information for the instance #}
[publish-caddy-information]
......@@ -395,6 +396,7 @@ slave-instance-information-list = {{ json_module.dumps(slave_instance_informatio
monitor-base-url = {{ monitor_base_url }}
[buildout]
extends = {{ common_profile }}
parts +=
slave-log-directories
{% for part in part_list %}
......@@ -409,9 +411,6 @@ parts +=
tunnel-6to4-base-nginx_http_port
tunnel-6to4-base-nginx_https_port
eggs-directory = {{ eggs_directory }}
develop-eggs-directory = {{ develop_eggs_directory }}
offline = true
cache-access = {{ cache_access }}
{% endif %}
software/caddy-frontend/templates/cached-virtualhost.conf.in
View file @ be08f156
......@@ -5,13 +5,13 @@
{%- set http_backend_host_list = [] %}
{%- set https_backend_host_list = [] %}
{%- for host in host_list %}
{%- do http_backend_host_list.append('http://%s:%s' % (host, cached_port)) %}
{%- do https_backend_host_list.append('http://%s:%s' % (host, ssl_cached_port)) %}
{%- do http_backend_host_list.append('http://%s:%s' % (host, slave_parameter['cached_port'])) %}
{%- do https_backend_host_list.append('http://%s:%s' % (host, slave_parameter['ssl_cached_port'])) %}
{%- endfor %}
# SSL-disabled backends
{{ http_backend_host_list|join(', ') }} {
bind {{ local_ipv4 }}
bind {{ slave_parameter['local_ipv4'] }}
# Compress the output
gzip
{%- if ssl_proxy_verify and 'ssl_proxy_ca_crt' in slave_parameter %}
......@@ -35,7 +35,7 @@
# SSL-enabled backends
{{ https_backend_host_list|join(', ') }} {
bind {{ local_ipv4 }}
bind {{ slave_parameter['local_ipv4'] }}
# Compress the output
gzip
{%- if ssl_proxy_verify and 'ssl_proxy_ca_crt' in slave_parameter %}
......
software/caddy-frontend/templates/caddy-wrapper.in deleted 100644 → 0
View file @ 121aff64
#!${dash-output:dash}
exec {{ caddy }} \
-conf {{ conf }} \
-log {{ log }} \
-http2=true \
-grace {{ grace }}s \
-disable-http-challenge \
-disable-tls-sni-challenge \
"$@"
software/caddy-frontend/templates/default-virtualhost.conf.in
View file @ be08f156
......@@ -17,13 +17,13 @@
{%- set http_host_list = [] %}
{%- set https_host_list = [] %}
{%- for host in host_list %}
{%- do http_host_list.append('http://%s:%s' % (host, http_port)) %}
{%- do https_host_list.append('https://%s:%s' % (host, https_port)) %}
{%- do http_host_list.append('http://%s:%s' % (host, slave_parameter['http_port'] )) %}
{%- do https_host_list.append('https://%s:%s' % (host, slave_parameter['https_port'] )) %}
{%- endfor %} {#- for host in host_list #}
# SSL enabled hosts
{{ https_host_list|join(', ') }} {
bind {{ local_ipv4 }}
bind {{ slave_parameter['local_ipv4'] }}
# Compress the output
gzip
{%- if ssl_proxy_verify and 'ssl_proxy_ca_crt' in slave_parameter %}
......@@ -144,7 +144,7 @@
# SSL-disabled hosts
{{ http_host_list|join(', ') }} {
bind {{ local_ipv4 }}
bind {{ slave_parameter['local_ipv4'] }}
# Compress the output
gzip
{%- if ssl_proxy_verify and 'ssl_proxy_ca_crt' in slave_parameter %}
......
software/caddy-frontend/templates/nginx-eventsource-slave.conf.in
View file @ be08f156
......@@ -21,7 +21,7 @@
# TODO-Caddy server {
# TODO-Caddy listen {{ local_ipv4 }}:{{ nginx_http_port }};
# TODO-Caddy listen {{ slave_parameter['local_ipv4'] }}:{{ slave_parameter['nginx_http_port'] }};
# TODO-Caddy
# TODO-Caddy server_name {{ slave_parameter.get('custom_domain') }};
# TODO-Caddy
......@@ -60,7 +60,7 @@
# TODO-Caddy
# TODO-Caddy server {
# TODO-Caddy listen {{ local_ipv4 }}:{{ nginx_https_port }} ssl;
# TODO-Caddy listen {{ slave_parameter['local_ipv4'] }}:{{ slave_parameter['nginx_https_port'] }} ssl;
# TODO-Caddy
# TODO-Caddy server_name {{ slave_parameter.get('custom_domain') }};
# TODO-Caddy
......
software/caddy-frontend/templates/nginx-notebook-slave.conf.in
View file @ be08f156
......@@ -5,8 +5,8 @@
{%- set https_upstream = https_url.split("/")[2] %}
# SSL-enabled
https://{{ slave_parameter.get('custom_domain') }}:{{ nginx_https_port }} {
bind {{ local_ipv4 }}
https://{{ slave_parameter.get('custom_domain') }}:{{ slave_parameter['nginx_https_port'] }} {
bind {{ slave_parameter['local_ipv4'] }}
# Compress the output
gzip
log / {{ slave_parameter.get('access_log') }} "{remote} {>REMOTE_USER} [{when}] \"{method} {uri} {proto}\" {status} {size} \"{>Referer}\" \"{>User-Agent}\" {latency_ms}"
......@@ -37,8 +37,8 @@ https://{{ slave_parameter.get('custom_domain') }}:{{ nginx_https_port }} {
}
# SSL-disabled
http://{{ slave_parameter.get('custom_domain') }}:{{ nginx_http_port }} {
bind {{ local_ipv4 }}
http://{{ slave_parameter.get('custom_domain') }}:{{ slave_parameter['nginx_http_port'] }} {
bind {{ slave_parameter['local_ipv4'] }}
# Compress the output
gzip
log / {{ slave_parameter.get('access_log') }} "{remote} {>REMOTE_USER} [{when}] \"{method} {uri} {proto}\" {status} {size} \"{>Referer}\" \"{>User-Agent}\" {latency_ms}"
......
software/caddy-frontend/templates/replicate-publish-slave-information.cfg.in
View file @ be08f156
......@@ -41,11 +41,8 @@ log-access-url = {{ json_module.dumps(slave_information.pop('log-access-urls', 1
{% endfor %}
[buildout]
extends = {{ common_profile }}
parts =
{% for part in part_list %}
{{ ' %s' % part }}
{% endfor %}
eggs-directory = {{ eggs_directory }}
develop-eggs-directory = {{ develop_eggs_directory }}
offline = true
\ No newline at end of file
{% endfor %}
\ No newline at end of file
software/caddy-frontend/templates/template-log-access.conf.in
View file @ be08f156
{% for slave, directory in slave_log_directory.iteritems() %}
https://[{{ global_ipv6 }}]:{{ https_port }}/{{ slave }}, https://{{ local_ipv4 }}:{{ https_port }}/{{ slave }} {
bind {{ local_ipv4 }}
root {{directory}}/
https://[{{ parameter_dict['global_ipv6'] }}]:{{ parameter_dict['https_port'] }}/{{ slave }}, https://{{ parameter_dict['local_ipv4'] }}:{{ parameter_dict['https_port'] }}/{{ slave }} {
bind {{ parameter_dict['local_ipv4'] }}
root {{ directory }}/
browse
tls {{ login_certificate }} {{ login_key }}
basicauth "{{ slave }}" {{ slave_password[slave] }} {
tls {{ parameter_dict['login_certificate'] }} {{ parameter_dict['login_key'] }}
basicauth "{{ slave }}" {{ slave_password[slave] | trim }} {
"Log Access {{ slave }}"
/
}
log / {{ access_log }} "{remote} {>REMOTE_USER} [{when}] \"{method} {uri} {proto}\" {status} {size} \"{>Referer}\" \"{>User-Agent}\" {latency_ms}"
errors {{ error_log }} {
* {{ not_found_file }}
log / {{ parameter_dict['access_log'] }} "{remote} {>REMOTE_USER} [{when}] \"{method} {uri} {proto}\" {status} {size} \"{>Referer}\" \"{>User-Agent}\" {latency_ms}"
errors {{ parameter_dict['error_log'] }} {
* {{ parameter_dict['not_found_file'] }}
}
}
{% endfor %}
software/caddy-frontend/test/test.py
View file @ be08f156
......@@ -758,6 +758,17 @@ class TestSlave(SlaveHttpFrontendTestCase, TestDataMixin):
os.path.join(
partition_path, 'etc', 'httpd-cors.cfg'), 'r').read().strip())
def test_promise_monitor_httpd_listening_on_tcp(self):
result = set([
subprocess.call(q) for q in glob.glob(
os.path.join(
self.instance_path, '*', 'etc', 'promise',
'monitor-httpd-listening-on-tcp'))])
self.assertEqual(
result,
set([0])
)
@skipIf(not IS_CADDY, 'Will NOT be covered on apache-frontend')
def test_slave_partition_state(self):
partition_path = self.getSlavePartitionPath()
......@@ -2780,3 +2791,42 @@ class TestMalformedBackenUrlSlave(SlaveHttpFrontendTestCase,
self.assertEqual(
parameter_dict, {}
)
class TestDefaultMonitorHttpdPort(SlaveHttpFrontendTestCase, TestDataMixin):
@classmethod
def getInstanceParameterDict(cls):
return {
'-frontend-1-state': 'stopped',
}
@classmethod
def getSlaveParameterDictDict(cls):
return {
'test': {
'url': cls.backend_url,
},
}
def test(self):
parameter_dict = self.slave_connection_parameter_dict_dict[
'test']
self.assertKeyWithPop('log-access-url', parameter_dict)
self.assertEqual(
parameter_dict,
{
'domain': 'test.None', 'replication_number': '1',
'url': 'http://test.None', 'site_url': 'http://test.None',
'secure_access': 'https://test.None', 'public-ipv4': None}
)
master_monitor_conf = open(os.path.join(
self.instance_path, 'TestDefaultMonitorHttpdPort-0', 'etc',
'monitor-httpd.conf')).read()
slave_monitor_conf = open(os.path.join(
self.instance_path, 'TestDefaultMonitorHttpdPort-1', 'etc',
'monitor-httpd.conf')).read()
self.assertTrue(
'Listen [%s]:8196' % (utils.GLOBAL_IPV6,) in master_monitor_conf)
self.assertTrue(
'Listen [%s]:8072' % (utils.GLOBAL_IPV6,) in slave_monitor_conf)
software/caddy-frontend/test/test_data/test.TestDefaultMonitorHttpdPort.test_file_list_run-CADDY.txt 0 → 100644
View file @ be08f156
TestDefaultMonitorHttpdPort-0/var/run/monitor/monitor-bootstrap.pid
\ No newline at end of file
software/caddy-frontend/test/test_data/test.TestDefaultMonitorHttpdPort.test_monitor_promise_list-CADDY.txt 0 → 100644
View file @ be08f156
TestDefaultMonitorHttpdPort-1/etc/monitor-promise/check-_test-error-log-last-day
TestDefaultMonitorHttpdPort-1/etc/monitor-promise/check-_test-error-log-last-hour
\ No newline at end of file
software/caddy-frontend/test/test_data/test.TestDefaultMonitorHttpdPort.test_promise_list-CADDY.txt 0 → 100644
View file @ be08f156
TestDefaultMonitorHttpdPort-0/etc/promise/check-free-disk-space
TestDefaultMonitorHttpdPort-0/etc/promise/monitor-http-frontend
TestDefaultMonitorHttpdPort-0/etc/promise/monitor-httpd-listening-on-tcp
TestDefaultMonitorHttpdPort-0/etc/promise/promise-monitor-httpd-is-process-older-than-dependency-set
TestDefaultMonitorHttpdPort-1/etc/promise/caddy-frontend-is-running-actual-software-release
TestDefaultMonitorHttpdPort-1/etc/promise/caddy_cached
TestDefaultMonitorHttpdPort-1/etc/promise/caddy_frontend_ipv4_http
TestDefaultMonitorHttpdPort-1/etc/promise/caddy_frontend_ipv4_https
TestDefaultMonitorHttpdPort-1/etc/promise/caddy_frontend_ipv6_http
TestDefaultMonitorHttpdPort-1/etc/promise/caddy_frontend_ipv6_https
TestDefaultMonitorHttpdPort-1/etc/promise/caddy_ssl_cached
TestDefaultMonitorHttpdPort-1/etc/promise/check-free-disk-space
TestDefaultMonitorHttpdPort-1/etc/promise/frontend-caddy-configuration-promise
TestDefaultMonitorHttpdPort-1/etc/promise/monitor-http-frontend
TestDefaultMonitorHttpdPort-1/etc/promise/monitor-httpd-listening-on-tcp
TestDefaultMonitorHttpdPort-1/etc/promise/nginx-configuration-promise
TestDefaultMonitorHttpdPort-1/etc/promise/nginx_frontend_ipv4_http
TestDefaultMonitorHttpdPort-1/etc/promise/nginx_frontend_ipv4_https
TestDefaultMonitorHttpdPort-1/etc/promise/nginx_frontend_ipv6_http
TestDefaultMonitorHttpdPort-1/etc/promise/nginx_frontend_ipv6_https
TestDefaultMonitorHttpdPort-1/etc/promise/promise-monitor-httpd-is-process-older-than-dependency-set
TestDefaultMonitorHttpdPort-1/etc/promise/promise-nginx-is-process-older-than-dependency-set
TestDefaultMonitorHttpdPort-1/etc/promise/re6st-connectivity
TestDefaultMonitorHttpdPort-1/etc/promise/trafficserver-cache-availability
TestDefaultMonitorHttpdPort-1/etc/promise/trafficserver-port-listening
\ No newline at end of file
software/caddy-frontend/test/test_data/test.TestDefaultMonitorHttpdPort.test_supervisor_state-CADDY.txt 0 → 100644
View file @ be08f156
TestDefaultMonitorHttpdPort-0:bootstrap-monitor EXITED
TestDefaultMonitorHttpdPort-0:certificate_authority-on-watch RUNNING
TestDefaultMonitorHttpdPort-0:crond RUNNING
TestDefaultMonitorHttpdPort-0:monitor-httpd-graceful EXITED
TestDefaultMonitorHttpdPort-0:monitor-httpd-on-watch EXITED
TestDefaultMonitorHttpdPort-1:6tunnel-26011-on-watch STOPPED
TestDefaultMonitorHttpdPort-1:6tunnel-26012-on-watch STOPPED
TestDefaultMonitorHttpdPort-1:6tunnel-4443-on-watch STOPPED
TestDefaultMonitorHttpdPort-1:6tunnel-8080-on-watch STOPPED
TestDefaultMonitorHttpdPort-1:6tunnel-8081-on-watch STOPPED
TestDefaultMonitorHttpdPort-1:6tunnel-9443-on-watch STOPPED
TestDefaultMonitorHttpdPort-1:bootstrap-monitor STOPPED
TestDefaultMonitorHttpdPort-1:certificate_authority-on-watch STOPPED
TestDefaultMonitorHttpdPort-1:crond-on-watch STOPPED
TestDefaultMonitorHttpdPort-1:frontend-caddy-safe-graceful STOPPED
TestDefaultMonitorHttpdPort-1:frontend-nginx-safe-graceful STOPPED
TestDefaultMonitorHttpdPort-1:frontend_caddy-on-watch STOPPED
TestDefaultMonitorHttpdPort-1:frontend_nginx-on-watch STOPPED
TestDefaultMonitorHttpdPort-1:monitor-httpd-graceful STOPPED
TestDefaultMonitorHttpdPort-1:monitor-httpd-on-watch STOPPED
TestDefaultMonitorHttpdPort-1:trafficserver-on-watch STOPPED
TestDefaultMonitorHttpdPort-1:trafficserver-reload STOPPED
watchdog:watchdog RUNNING
\ No newline at end of file
software/kvm/instance-kvm-cluster.cfg.jinja2.in
View file @ be08f156
......@@ -91,7 +91,7 @@ config-httpd-port = {{ dumps(kvm_parameter_dict.get('httpd-port', 8081)) }}
config-disable-ansible-promise = {{ dumps(kvm_parameter_dict.get('disable-ansible-promise', False)) }}
config-monitor-cors-domains = {{ slapparameter_dict.get('monitor-cors-domains', 'monitor.app.officejs.com') }}
config-monitor-username = ${monitor-instance-parameter:username}
config-monitor-password = ${monitor-htpasswd:passwd}
config-monitor-password = ${publish-early:monitor-password}
# Enable disk wipe options
{% if kvm_parameter_dict.get('wipe-disk-ondestroy', False) -%}
config-wipe-disk-ondestroy = True
......@@ -238,6 +238,11 @@ mode = {{ mode }}
{{ writefile('cluster-data-content', '${directory:webroot}/${hash-code:passwd}/data', slapparameter_dict.get('cluster-data', ''), '700') }}
{% endif -%}
[publish-early]
recipe = slapos.cookbook:publish-early
-init =
monitor-password monitor-htpasswd:passwd
[monitor-instance-parameter]
monitor-httpd-port = 8060
cors-domains = {{ slapparameter_dict.get('monitor-cors-domains', 'monitor.app.officejs.com') }}
......@@ -260,6 +265,8 @@ recipe = slapos.cookbook:publish
{% endfor %}
{% set monitor_interface_url = slapparameter_dict.get('monitor-interface-url', 'https://monitor.app.officejs.com') -%}
monitor-setup-url = {{ monitor_interface_url }}/#page=settings_configurator&url=${monitor-publish-parameters:monitor-url}&username=${monitor-publish-parameters:monitor-user}&password=${monitor-publish-parameters:monitor-password}
monitor-password = ${publish-early:monitor-password}
monitor-user = ${monitor-publish-parameters:monitor-user}
{% do part_list.append('monitor-base') -%}
[buildout]
......@@ -273,7 +280,6 @@ parts =
httpd-promise
publish-connection-information
directory-doc
monitor-htpasswd
# Complete parts with sections
{{ part_list | join('\n ') }}
......
software/kvm/instance-kvm-resilient.cfg.jinja2
View file @ be08f156
......@@ -21,7 +21,7 @@ offline = true
# += because we need to take up parts (like instance-custom, slapmonitor etc) from the profile we extended
parts +=
monitor-htpasswd
publish-early
{{ parts.replicate("kvm", backup_amount) }}
publish-connection-information
kvm-frontend-url-promise
......@@ -37,13 +37,18 @@ storage-path = ${directory:etc}/.monitor_user
bytes = 8
username = admin
[publish-early]
recipe = slapos.cookbook:publish-early
-init =
monitor-password monitor-htpasswd:passwd
# XXX Monitoring Main Instane
[monitor-instance-parameter]
monitor-httpd-port = 8160
cors-domains = {{ monitor_parameter.get('monitor-cors-domains', '') }}
{% do monitor_parameter.__setitem__('monitor-username', slapparameter_dict.get('monitor-username', 'admin'))%}
{% do monitor_parameter.__setitem__('monitor-password', slapparameter_dict.get('monitor-password', '${monitor-htpasswd:passwd}'))%}
{% do monitor_parameter.__setitem__('monitor-password', slapparameter_dict.get('monitor-password', '${publish-early:monitor-password}'))%}
{% endif -%}
{{ replicated.replicate("kvm", backup_amount, "kvm-export", "kvm-import", slapparameter_dict=slapparameter_dict, monitor_parameter_dict=monitor_dict) }}
......@@ -68,9 +73,11 @@ recipe = slapos.cookbook:publish
backend-url = ${request-kvm:connection-backend-url}
url = ${request-kvm:connection-url}
ipv6 = ${request-kvm:connection-ip}
monitor-password = ${publish-early:monitor-password}
monitor-user = ${monitor-publish-parameters:monitor-user}
{% if monitor_dict -%}
monitor-base-url = ${monitor-publish-parameters:monitor-base-url}
monitor-setup-url = {{ monitor_interface_url }}/#page=settings_configurator&url=${monitor-publish-parameters:monitor-url}&username=${monitor-publish-parameters:monitor-user}&password=${monitor-publish-parameters:monitor-password}
monitor-setup-url = {{ monitor_interface_url }}/#page=settings_configurator&url=${monitor-publish-parameters:monitor-url}&username=${monitor-publish-parameters:monitor-user}&password=${publish-early:monitor-password}
{% endif -%}
[kvm-frontend-url-promise]
......
software/kvm/software.cfg
View file @ be08f156
......@@ -108,7 +108,7 @@ recipe = hexagonit.recipe.download
ignore-existing = true
url = ${:_profile_base_location_}/instance-kvm-cluster.cfg.jinja2.in
mode = 644
md5sum = d9fe920d31f1ef0e377aa768ccd24f4c
md5sum = 6d165aec7d236ea3944765236d11940f
download-only = true
on-update = true
......@@ -117,7 +117,7 @@ recipe = hexagonit.recipe.download
ignore-existing = true
url = ${:_profile_base_location_}/instance-kvm-resilient.cfg.jinja2
mode = 644
md5sum = 1095968487282784a735735aa1b37d35
md5sum = a5fd0cbe6be757d57c8b6903bb7a1d8b
download-only = true
on-update = true
......
software/re6stnet/instance-re6stnet-input-schema.json
View file @ be08f156
......@@ -72,6 +72,12 @@
"description": "Specify that tunnels should be encrypted.",
"type": "boolean",
"default": false
},
"same-country": {
"title": "Same Country",
"description": "Prevent tunnelling accross borders of listed countries",
"type": "string",
"default": ""
}
}
}
\ No newline at end of file
software/re6stnet/instance-re6stnet.cfg.in
View file @ be08f156
......@@ -125,6 +125,7 @@ max-clients = {{ slapparameter_dict.get('max-clients', 0) }}
hello = {{ slapparameter_dict.get('hello', 15) }}
min-protocol = {{ slapparameter_dict.get('min-protocol', -1) }}
encrypt = {{ slapparameter_dict.get('encrypt', 'False') }}
same-country = {{ slapparameter_dict.get('same-country', '') }}
[re6st-registry-conf]
recipe = slapos.recipe.template:jinja2
......
software/re6stnet/re6st-registry.conf.in
View file @ be08f156
......@@ -25,4 +25,7 @@ encrypt
{% endif -%}
{% if parameter_dict.get('max-clients') != '0' -%}
max-clients {{ parameter_dict['max-clients'] }}
{% endif -%}
{% if parameter_dict.get('same-country') -%}
same-country {{ parameter_dict['same-country'] }}
{% endif -%}
\ No newline at end of file
software/re6stnet/software.cfg
View file @ be08f156
......@@ -87,7 +87,7 @@ extra-context =
[template-re6stnet]
< = download-base
filename = instance-re6stnet.cfg.in
md5sum = 6f28b611a0e2415768238f5e8d29d36e
md5sum = 8c167f2adb2ed36aeaff773f59214981
[template-apache-conf]
< = download-base
......@@ -97,7 +97,7 @@ md5sum = d64cafda1139b740a49a9f5e30a1b57b
[template-re6st-registry-conf]
< = download-base
filename = re6st-registry.conf.in
md5sum = 5dc218f887faeffc466e41c7d6191e49
md5sum = b85375cd45c5f2fb0d68e449ae70e2a1
[template-wrapper]
< = download-base
......
software/slaprunner/buildout.hash.cfg
View file @ be08f156
......@@ -38,7 +38,7 @@ md5sum = 1a812a06cc02bb11636009f4ec043d54
[template-resilient]
filename = instance-resilient.cfg.jinja2
md5sum = 8ed180de711d207a540d0acb539b2536
md5sum = bed1c457aa9e54a59b64d167bdafe970
[template_nginx_conf]
filename = nginx_conf.in
......
software/slaprunner/instance-resilient.cfg.jinja2
View file @ be08f156
......@@ -28,7 +28,7 @@ offline = true
# += because we need to take up parts (like instance-custom, slapmonitor etc) from the profile we extended
parts +=
monitor-htpasswd
publish-early
{{ parts.replicate("runner", number_of_instances + 1) }}
publish-connection-information
......@@ -38,8 +38,13 @@ storage-path = ${directory:etc}/.monitor_user
bytes = 8
username = admin
[publish-early]
recipe = slapos.cookbook:publish-early
-init =
init-password monitor-htpasswd:passwd
{% do monitor_parameter.__setitem__('monitor-username', slapparameter_dict.get('monitor-username', 'admin'))%}
{% do monitor_parameter.__setitem__('monitor-password', slapparameter_dict.get('monitor-password', '${monitor-htpasswd:passwd}'))%}
{% do monitor_parameter.__setitem__('monitor-password', slapparameter_dict.get('monitor-password', '${publish-early:init-password}'))%}
{{ replicated.replicate("runner", number_of_instances + 1, "runner-export", "runner-import", slapparameter_dict=slapparameter_dict, monitor_parameter_dict=monitor_dict) }}
......@@ -60,7 +65,7 @@ recipe = slapos.cookbook:publish
backend-url = ${request-runner:connection-backend-url}
url = ${request-runner:connection-url}
init-user = ${request-runner:connection-init-user}
init-password = ${request-runner:connection-init-password}
init-password = ${publish-early:init-password}
ssh-command = ${request-runner:connection-ssh-command}
webdav-url = ${request-runner:connection-webdav-url}
public-url = ${request-runner:connection-public-url}
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment
GitLab Nexedi Edition | About GitLab | About Nexedi | 沪ICP备2021021310号-2 | 沪ICP备2021021310号-7