Skip to content

slapos
slapos
Commit 6cc11cdf authored by Łukasz Nowak's avatar Łukasz Nowak
Browse files
Options

caddy-frontend: Put haproxy just before the backend 

This is needed in order to provide future support for client certificates
to the backend.

Also it means that haproxy is used in all cases, with or without cache, and as
a result the "cached" version of caddy is dropped.

fixup! caddy-frontend: Put haproxy just before the backend
parent 00288e68
Expand all Show whitespace changes
Inline Side-by-side
Showing with 416 additions and 366 deletions
software/caddy-frontend/README.caddy_frontend.rst
View file @ 6cc11cdf
......@@ -472,11 +472,22 @@ Instantiating caddy-frontend results with a cluster in various partitions:
* kedifa (contains kedifa server)
* caddy-frontend-N which contains the running processes to serve sites - this partition can be replicated by ``-frontend-quantity`` parameter
So it means sites are served in `caddy-frontend-N` partition, and this partition is structured as:
It means sites are served in `caddy-frontend-N` partition, and this partition is structured as:
* Caddy serving the browser
* (optional) Apache Traffic Server for caching
* Caddy connected to the backend
* Caddy serving the browser [frontend-caddy]
* (optional) Apache Traffic Server for caching [ats]
* Haproxy as a way to communicate to the backend [backend-haproxy]
* some other additional tools (6tunnel, monitor, etc)
In case of slaves without cache (`enable_cache = False`) the request will travel as follows::
frontend-caddy --> backend-haproxy --> backend
In case of slaves using cache (`enable_cache = True`) the request will travel as follows::
frontend-caddy --> ats --> backend-haproxy --> backend
Usage of Haproxy as a relay to the backend allows much better control of the backend, removes the hassle of checking the backend from Caddy and allows future developments like client SSL certificates to the backend or even health checks.
Kedifa implementation
---------------------
......
software/caddy-frontend/buildout.hash.cfg
View file @ 6cc11cdf
......@@ -14,7 +14,7 @@
# not need these here).
[template]
filename = instance.cfg.in
md5sum = 20f7a925e686949092823595c79a0523
md5sum = 3214dec50a312d5acc27f7947fcc6f0c
[template-common]
filename = instance-common.cfg.in
......@@ -22,7 +22,7 @@ md5sum = c801b7f9f11f0965677c22e6bbe9281b
[template-apache-frontend]
filename = instance-apache-frontend.cfg.in
md5sum = 0851faa528eb4f21330a6f23f77dea7f
md5sum = c0325f8cabba57b399c3f5ed2c6a01c4
[template-caddy-replicate]
filename = instance-apache-replicate.cfg.in
......@@ -30,7 +30,7 @@ md5sum = 6d7113ebf0c46b0e4c72c128ebb647db
[template-slave-list]
_update_hash_filename_ = templates/apache-custom-slave-list.cfg.in
md5sum = 9da1616d203e4909af37e658aa923d95
md5sum = c72c8d6d0189aa19a743c96e8d7215cf
[template-replicate-publish-slave-information]
_update_hash_filename_ = templates/replicate-publish-slave-information.cfg.in
......@@ -38,7 +38,7 @@ md5sum = 7e3ee70c447f8203273d78f66ab519c3
[template-caddy-frontend-configuration]
_update_hash_filename_ = templates/Caddyfile.in
md5sum = f0faf6d2e6c187df7e25bf717676f9df
md5sum = 2503056e35463e045db3329bb8b6fae8
[caddy-backend-url-validator]
filename = templates/caddy-backend-url-validator.in
......@@ -50,11 +50,11 @@ md5sum = f20d6c3d2d94fb685f8d26dfca1e822b
[template-default-slave-virtualhost]
_update_hash_filename_ = templates/default-virtualhost.conf.in
md5sum = a72e9056eeda3c7c794f6f6560056380
md5sum = 82c3908c9f6346356179646211058bf5
[template-cached-slave-virtualhost]
_update_hash_filename_ = templates/cached-virtualhost.conf.in
md5sum = e839ca3cb308f7fcdfa06c2f1b95e93f
[template-backend-haproxy-configuration]
_update_hash_filename_ = templates/backend-haproxy.cfg.in
md5sum = eb517d1f051f0631744a512af84b9a3b
[template-log-access]
_update_hash_filename_ = templates/template-log-access.conf.in
......@@ -94,7 +94,7 @@ md5sum = 061cc244558fd3af2b6bacf17cae5555
[template-validate-script]
_update_hash_filename_ = templates/validate-script.sh.in
md5sum = f26e11574f266c7437c9c89e3c93825a
md5sum = 53e5d7ba2827bff003051f74f24ffe4f
[template-configuration-state-script]
_update_hash_filename_ = templates/configuration-state-script.sh.in
......
software/caddy-frontend/common.cfg
View file @ 6cc11cdf
......@@ -10,6 +10,7 @@ extends =
../../component/trafficserver/buildout.cfg
../../component/6tunnel/buildout.cfg
../../component/xz-utils/buildout.cfg
../../component/haproxy/buildout.cfg
../../stack/caucase/buildout.cfg
# Monitoring stack (keep on bottom)
......@@ -94,6 +95,7 @@ bin_directory = ${buildout:bin-directory}
sixtunnel = ${6tunnel:location}
caddy = ${caddy:output}
caddy_location = ${caddy:location}
haproxy_executable = ${haproxy:location}/sbin/haproxy
curl = ${curl:location}
dash = ${dash:location}
gzip = ${gzip:location}
......@@ -108,7 +110,7 @@ kedifa-csr = ${:bin_directory}/kedifa-csr
xz_location = ${xz-utils:location}
monitor_template = ${monitor-template:output}
template_cached_slave_virtualhost = ${template-cached-slave-virtualhost:target}
template_backend_haproxy_configuration = ${template-backend-haproxy-configuration:target}
template_caddy_frontend_configuration = ${template-caddy-frontend-configuration:target}
template_graceful_script = ${template-graceful-script:target}
template_validate_script = ${template-validate-script:target}
......@@ -184,7 +186,7 @@ mode = 640
[template-default-slave-virtualhost]
<=download-template
[template-cached-slave-virtualhost]
[template-backend-haproxy-configuration]
<=download-template
[template-log-access]
......
software/caddy-frontend/instance-caddy-input-schema.json
View file @ 6cc11cdf
......@@ -66,16 +66,16 @@
"title": "Test Verification URL",
"type": "string"
},
"proxy-try-duration": {
"timeout-backend-connect": {
"default": 5,
"description": "A time during which Caddy will try to establish connection with a backend. Setting it to 0 will result with immediate return of 502 EOF error to the browser, even if it would be possible to (re)connect to the backend during few moments. More info in https://caddyserver.com/docs/proxy try_durtion.",
"description": "A time for wait for the backend to be connected..",
"title": "Duration in seconds of trying a backend",
"type": "integer"
},
"proxy-try-interval": {
"default": 250,
"description": "How often Caddy will try to establish connection with a backend during proxy-try-duration. More info in https://caddyserver.com/docs/proxy try_interval",
"title": "Interval in milliseconds of tries during proxy-try-duration",
"timeout-backend-connect-retries": {
"default": 3,
"description": "Amount of retries to connect to the backend. The amount of timeout-backend-connect*timeout-backend-connect-retries seconds will be spent to connect to the backend.",
"title": "Amount of retries to connect to the backend.",
"type": "integer"
},
"automatic-internal-kedifa-caucase-csr": {
......
software/caddy-frontend/instance-slave-caddy-input-schema.json
View file @ 6cc11cdf
......@@ -195,6 +195,16 @@
"title": "type:zope virtualhostroot-https-port",
"type": "integer"
},
"timeout-backend-connect": {
"description": "A time for wait for the backend to be connected..",
"title": "Duration in seconds of trying a backend",
"type": "integer"
},
"timeout-backend-connect-retries": {
"description": "Amount of retries to connect to the backend. The amount of timeout-backend-connect*timeout-backend-connect-retries seconds will be spent to connect to the backend.",
"title": "Amount of retries to connect to the backend.",
"type": "integer"
},
"ciphers": {
"description": "List of ciphers. Empty defaults to cluster list of ciphers, which by default are Caddy list of ciphers. See https://caddyserver.com/docs/tls for more information.",
"title": "Ordered space separated list of ciphers",
......
software/caddy-frontend/instance.cfg.in
View file @ 6cc11cdf
......@@ -117,5 +117,7 @@ configuration.ciphers =
configuration.request-timeout = 600
configuration.mpm-graceful-shutdown-timeout = 5
configuration.frontend-name =
configuration.proxy-try-duration = 5
configuration.proxy-try-interval = 250
configuration.timeout-backend-connect = 5
configuration.timeout-backend-connect-retries = 3
configuration.backend-haproxy-http-port = 21080
configuration.backend-haproxy-https-port = 21443
software/caddy-frontend/templates/Caddyfile.in
View file @ 6cc11cdf
......@@ -2,10 +2,8 @@
import {{frontend_configuration.get('log-access-configuration')}}
import {{ slave_configuration_directory }}/*.conf
import {{ slave_with_cache_configuration_directory }}/*.conf
{% for port in [https_port] %}
:{{ port }} {
:{{ https_port }} {
tls {{ master_certificate }} {{ master_certificate }}
bind {{ local_ipv4 }}
status 404 /
......@@ -17,10 +15,8 @@ import {{ slave_with_cache_configuration_directory }}/*.conf
* {{ not_found_file }}
}
}
{%- endfor %}
{% for port in [http_port, cached_port, ssl_cached_port] %}
:{{ port }} {
:{{ http_port }} {
bind {{ local_ipv4 }}
status 404 /
log / {{ access_log }} "{remote} - {>REMOTE_USER} [{when}] \"{method} {uri} {proto}\" {status} {size} \"{>Referer}\" \"{>User-Agent}\" {latency_ms}" {
......@@ -31,7 +27,6 @@ import {{ slave_with_cache_configuration_directory }}/*.conf
* {{ not_found_file }}
}
}
{%- endfor %}
# Access to server-status Caddy-style
https://[{{ global_ipv6 }}]:{{ https_port }}/server-status, https://{{ local_ipv4 }}:{{ https_port }}/server-status {
......
software/caddy-frontend/templates/backend-haproxy.cfg.in 0 → 100644
View file @ 6cc11cdf
# NON PROD CONFIG
global
maxconn 4096
pidfile {{ configuration['pid-file'] }}
# master-worker is compatible with foreground with process management
master-worker
log stderr local0
defaults
log global
mode http
option httplog
option dontlognull
retries 1
option redispatch
cookie SERVERID rewrite
balance roundrobin
stats uri /haproxy
stats realm Global\ statistics
timeout queue 60s
option httpclose
{%- macro frontend_entry(slave_instance, scheme) %}
{%- set host_list = slave_instance.get('server-alias', '').split() %}
{%- if slave_instance.get('custom_domain') not in host_list %}
{%- do host_list.append(slave_instance.get('custom_domain')) %}
{%- endif %}
{%- for host in host_list %}
{%- if host.startswith('*.') %}
{#- hdr_sub has to be used, as anyway something.example.com shall match *.example.com[:port], with optional port #}
acl is_{{ slave_instance['slave_reference'] }} hdr_sub(host) -i {{ host[2:] }}
{%- else %}
acl is_{{ slave_instance['slave_reference'] }} hdr_dom(host) -i {{ host }}
{%- endif %}
{%- endfor %}
use_backend {{ slave_instance['slave_reference'] }}-{{ scheme }} if is_{{ slave_instance['slave_reference'] }}
{%- endmacro %}
frontend http-backend
bind {{ configuration['local-ipv4'] }}:{{ configuration['http-port'] }}
{%- for slave_instance in backend_slave_list %}
{{ frontend_entry(slave_instance, 'http') }}
{%- endfor %}
frontend https-backend
bind {{ configuration['local-ipv4'] }}:{{ configuration['https-port'] }}
{%- for slave_instance in backend_slave_list %}
{{ frontend_entry(slave_instance, 'https') }}
{%- endfor %}
{%- for slave_instance in backend_slave_list %}
{%- for (scheme, prefix) in [('http', 'http_backend'), ('https', 'https_backend')] %}
{%- set info_dict = slave_instance[prefix] %}
{%- if info_dict['scheme'] == 'https' %}
{%- set ssl = ['ssl verify'] %}
{%- set path_to_ssl_proxy_ca_crt = slave_instance.get('path_to_ssl_proxy_ca_crt') %}
{%- if slave_instance['ssl_proxy_verify'] %}
{%- if path_to_ssl_proxy_ca_crt %}
{%- do ssl.append('required ca-file %s' % (path_to_ssl_proxy_ca_crt,)) %}
{%- else %}
{#- Backend SSL shall be verified, but not CA provided, disallow connection #}
{#- Simply dropping hostname from the dict will result with ignoring it... #}
{%- do info_dict.__setitem__('hostname', '') %}
{%- endif %}
{%- else %}
{%- do ssl.append('none') %}
{%- endif %}
{%- set ssl = ' '.join(ssl) %}
{%- else %}
{%- set ssl = '' %}
{%- endif %}
backend {{ slave_instance['slave_reference'] }}-{{ scheme }}
timeout server {{ slave_instance['request-timeout'] }}s
timeout client {{ slave_instance['request-timeout'] }}s
timeout connect {{ slave_instance['timeout-backend-connect'] }}s
retries {{ slave_instance['timeout-backend-connect-retries'] }}s
{%- set hostname = info_dict['hostname'] %}
{%- set port = info_dict['port'] %}
{%- set path = info_dict['path'].rstrip('/') %}
{%- if hostname and port %}
server backend {{ hostname }}:{{ port }} {{ ssl }}
{%- if path %}
http-request set-path {{ path }}%[path]
{%- endif %}
{%- endif %}
{%- endfor %}
{%- endfor %}
software/caddy-frontend/templates/cached-virtualhost.conf.in deleted 100644 → 0
View file @ 00288e68
{%- set TRUE_VALUES = ['y', 'yes', '1', 'true'] %}
{%- set server_alias_list = slave_parameter.get('server-alias', '').split() %}
{%- set ssl_proxy_verify = ('' ~ slave_parameter.get('ssl-proxy-verify', '')).lower() in TRUE_VALUES %}
{%- set host_list = [] %}
{%- for host in [slave_parameter.get('custom_domain')] + server_alias_list %}
{%- if host not in host_list %}
{%- do host_list.append(host) %}
{%- endif %}
{%- endfor %}
{%- set http_backend_host_list = [] %}
{%- set https_backend_host_list = [] %}
{%- for host in host_list %}
{%- do http_backend_host_list.append('http://%s:%s' % (host, slave_parameter['cached_port'])) %}
{%- do https_backend_host_list.append('http://%s:%s' % (host, slave_parameter['ssl_cached_port'])) %}
{%- endfor %}
# SSL-disabled backends
{{ http_backend_host_list|join(', ') }} {
bind {{ slave_parameter['local_ipv4'] }}
# Rewrite part
proxy / {{ slave_parameter.get('backend_url', '') }} {
try_duration {{ slave_parameter['proxy_try_duration'] }}s
try_interval {{ slave_parameter['proxy_try_interval'] }}ms
header_upstream Host {host}
{# header_upstream -X-Forwarded-For - caddy behaviour while removing and setting header is unstable, so for now original header has to be kept, even if in that case it comes from after ATS caddy itself #}
header_upstream X-Forwarded-For {>X-Forwarded-For-Real}
header_upstream -X-Forwarded-For-Real
timeout {{ slave_parameter['request_timeout'] }}s
{%- if ssl_proxy_verify %}
{%- if 'path_to_ssl_proxy_ca_crt' in slave_parameter %}
ca_certificates {{ slave_parameter['path_to_ssl_proxy_ca_crt'] }}
{%- endif %}
{%- else %}
insecure_skip_verify
{%- endif %}
}
log / {{ slave_parameter.get('access_log_cache_direct') }} "{remote} - {>REMOTE_USER} [{when}] \"{method} {uri} {proto}\" {status} {size} \"{>Referer}\" \"{>User-Agent}\" {latency_ms}" {
rotate_size 0
}
errors {{ slave_parameter.get('error_log_cache_direct') }} {
rotate_size 0
}
}
# SSL-enabled backends
{{ https_backend_host_list|join(', ') }} {
bind {{ slave_parameter['local_ipv4'] }}
proxy / {{ slave_parameter.get('https_backend_url', '') }} {
try_duration {{ slave_parameter['proxy_try_duration'] }}s
try_interval {{ slave_parameter['proxy_try_interval'] }}ms
header_upstream Host {host}
{# header_upstream -X-Forwarded-For - caddy behaviour while removing and setting header is unstable, so for now original header has to be kept, even if in that case it comes from after ATS caddy itself #}
header_upstream X-Forwarded-For {>X-Forwarded-For-Real}
header_upstream -X-Forwarded-For-Real
timeout {{ slave_parameter['request_timeout'] }}s
{%- if ssl_proxy_verify %}
{%- if 'path_to_ssl_proxy_ca_crt' in slave_parameter %}
ca_certificates {{ slave_parameter['path_to_ssl_proxy_ca_crt'] }}
{%- endif %}
{%- else %}
insecure_skip_verify
{%- endif %}
}
log / {{ slave_parameter.get('access_log_cache_direct') }} "{remote} - {>REMOTE_USER} [{when}] \"{method} {uri} {proto}\" {status} {size} \"{>Referer}\" \"{>User-Agent}\" {latency_ms}" {
rotate_size 0
}
errors {{ slave_parameter.get('error_log_cache_direct') }} {
rotate_size 0
}
}
software/caddy-frontend/templates/default-virtualhost.conf.in
View file @ 6cc11cdf
......@@ -9,7 +9,6 @@
{%- endif %} {#- if prefer_gzip #}
{%- set server_alias_list = slave_parameter.get('server-alias', '').split() %}
{%- set enable_h2 = slave_parameter['global_disable_http2'].lower() not in TRUE_VALUES and slave_parameter.get('enable-http2', slave_parameter['enable_http2_by_default']).lower() in TRUE_VALUES %}
{%- set ssl_proxy_verify = slave_parameter.get('ssl-proxy-verify', '').lower() in TRUE_VALUES %}
{%- set disabled_cookie_list = slave_parameter.get('disabled-cookie-list', '').split() %}
{%- set https_only = slave_parameter.get('https-only', 'true').lower() in TRUE_VALUES %}
{%- set slave_type = slave_parameter.get('type', '') %}
......@@ -41,31 +40,18 @@
{%- endif %}
{%- macro proxy_header() %}
try_duration {{ slave_parameter['proxy_try_duration'] }}s
try_interval {{ slave_parameter['proxy_try_interval'] }}ms
timeout {{ slave_parameter['request_timeout'] }}s
{%- if ssl_proxy_verify %}
{%- if 'path_to_ssl_proxy_ca_crt' in slave_parameter %}
ca_certificates {{ slave_parameter['path_to_ssl_proxy_ca_crt'] }}
{%- endif %} {#- if 'path_to_ssl_proxy_ca_crt' in slave_parameter #}
{%- else %} {#- if ssl_proxy_verify #}
insecure_skip_verify
{%- endif %} {#- if ssl_proxy_verify #}
timeout {{ slave_parameter['request-timeout'] }}s
# force reset of X-Forwarded-For
header_upstream X-Forwarded-For {remote}
{%- if enable_cache %}
# provide a header for other components
header_upstream X-Forwarded-For-Real {remote}
{%- endif %}
{%- endmacro %} {# proxy_header #}
{%- for tls in [True, False] %}
{%- if tls %}
{%- set backend_url = slave_parameter.get('https-url', slave_parameter.get('url', '')).rstrip('/') %}
{%- set backend_url = slave_parameter.get('backend-https-url', slave_parameter.get('backend-http-url')) %}
# SSL enabled hosts
{{ https_host_list|join(', ') }} {
{%- else %}
{%- set backend_url = slave_parameter.get('url', '').rstrip('/') %}
{%- set backend_url = slave_parameter['backend-http-url'] %}
# SSL-disabled hosts
{{ http_host_list|join(', ') }} {
{%- endif %}
......@@ -178,11 +164,13 @@
{%- endif %}
} {# rewrite #}
{%- endif %} {#- if prefer_gzip #}
{%- elif slave_type == 'redirect' and backend_url %} {#- if slave_type == 'zope' and backend_url #}
{%- elif slave_type == 'redirect' %}
{%- if backend_url %}
# Redirect configuration
redir 302 {
/ {{ backend_url }}{rewrite_uri}
} {# redir #}
}
{%- endif %}
{%- elif slave_type == 'notebook' %}
proxy / {{ backend_url }} {
{{ proxy_header() }}
......@@ -204,6 +192,8 @@
{{ proxy_header() }}
{%- if websocket_transparent %}
transparent
{%- else %}
header_upstream Host {host}
{%- endif %}
}
{%- for websocket_path in websocket_path_list %}
......@@ -212,6 +202,8 @@
websocket
{%- if websocket_transparent %}
transparent
{%- else %}
header_upstream Host {host}
{%- endif %}
}
{%- endfor %}
......@@ -221,6 +213,8 @@
websocket
{%- if websocket_transparent %}
transparent
{%- else %}
header_upstream Host {host}
{%- endif %}
}
{%- endif %}
......
software/caddy-frontend/templates/validate-script.sh.in
View file @ 6cc11cdf
......@@ -10,10 +10,10 @@ if [ -f $LAST_STATE_FILE ] ; then
old_found=$(find $LAST_STATE_FILE -mmin +120 | wc -l)
fi
if [ "$old_found" -eq 1 ] || {{ caddy_configuration_state }} ; then
if [ "$old_found" -eq 1 ] || {{ configuration_state_command }} ; then
# do not catch errors during validation
set +e
{{ wrapper }} -validate
{{ validate_command }}
echo $? > $LAST_STATE_FILE
set -e
fi
......
software/caddy-frontend/test/test.py
View file @ 6cc11cdf
This diff is collapsed.
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment
GitLab Nexedi Edition | About GitLab | About Nexedi | 沪ICP备2021021310号-2 | 沪ICP备2021021310号-7