Commit af7a0208 authored by Jérome Perrin's avatar Jérome Perrin

ERP5: Test balancer partition and use caucase certificate for balancer

Revert f8f72a17 ([erp5] don't use caucase generated certificate for now, 2019-03-12) since nothing prevents us drom using caucase certificate now.
 
Use [managed resources](slapos.core!259) to simplify existing tests and introduce tests for:

## Access Log

 - [x] balancer partition should produce logs in apache "combined" log format with microsecond timing of requests.
 - [x] these logs should be rotated daily
 - [x] an [apachedex](https://lab.nexedi.com/nexedi/apachedex) report is ran on these logs daily.

## Balancing

 - [x] requests are balanced to multiple backends using round-robin algorithm
 - [x] if backend is down it is excluded
 - [x] a "sticky cookie" is used so that clients are associated to the same backend
    - [x] the cookie is set by balancer
    - [x] when client comes with a cookie it "sticks" on the associated backend
    - [x] if "sticked" backend is down, another backend will be used

## Content-Encoding

 - [x] balancer encodes responses in gzip for some configured content types.

## HTTP

 - [x] Server uses HTTP/1.1 or more and keep connection with clients

## TLS (server certificate)

In this MR we also change apache to use a caucase managed certificate and add test coverage for:

 - [x] balancer listen on https with a certificate that can be verified using the CA from caucase.
 - [x] balancer uses the new certificate when its own certificate is renewed.

But we don't add support for:
 -  ~~balancer can be instantiated with a certificate and key passed as SlapOS request parameters (code [here](https://lab.nexedi.com/nexedi/slapos/blob/757c1a4ddee93659d5e2649e4252d87bf9494566/stack/erp5/instance-balancer.cfg.in#L208-213))~~ this use case is the job of caucase, so we no longer support this.

## TLS (client certificate)
 - [x] balancer verifies frontend certificates from frontend caucases ( also tested in "Forwarded-For" section )
 - [x] if frontend provided a verified certificate, balancer set `remote-user` header
 - [x] balancer updates CRL from caucases ( `caucase-updater-housekeeper` )
 - (NOT TESTED) balancer updates CA certificate from caucase ( `caucase-updater-housekeeper` ). Since this is would be complex to test and basic functionality of `caucase-updater-housekeeper` for frontend caucases is covered by CRL test, we don't test this for simplicity.

## "Forwarded-For" header

This was also covered by existing tests:  

 - [x] balancer set `X-Forwarded-For` header when frontend certificate can be verified
 - [x] balancer strips existing `X-Forwarded-For`

## Integration with the rest of ERP5 software release

This was also covered by existing tests:  

- [x] The https URL of each Zope family is published and replies properly
- [x] Some https URLs are generated for `runUnitTest`, so that test run with an https certificate. This is also covered by regular ERP5 functional tests.

See merge request !840
parents babec085 f1173ea5
...@@ -33,7 +33,7 @@ with open("README.md") as f: ...@@ -33,7 +33,7 @@ with open("README.md") as f:
setup(name=name, setup(name=name,
version=version, version=version,
description="Test for SlapOS' ERP5 software releae", description="Test for SlapOS' ERP5 software release",
long_description=long_description, long_description=long_description,
long_description_content_type='text/markdown', long_description_content_type='text/markdown',
maintainer="Nexedi", maintainer="Nexedi",
...@@ -50,8 +50,9 @@ setup(name=name, ...@@ -50,8 +50,9 @@ setup(name=name,
'mysqlclient', 'mysqlclient',
'backports.lzma', 'backports.lzma',
'cryptography', 'cryptography',
'pexpect',
'pyOpenSSL', 'pyOpenSSL',
'typing; python_version<"3"',
], ],
zip_safe=True,
test_suite='test', test_suite='test',
) )
This diff is collapsed.
...@@ -31,6 +31,7 @@ import glob ...@@ -31,6 +31,7 @@ import glob
import urlparse import urlparse
import socket import socket
import time import time
import tempfile
import psutil import psutil
import requests import requests
...@@ -43,7 +44,7 @@ setUpModule # pyflakes ...@@ -43,7 +44,7 @@ setUpModule # pyflakes
class TestPublishedURLIsReachableMixin(object): class TestPublishedURLIsReachableMixin(object):
"""Mixin that checks that default page of ERP5 is reachable. """Mixin that checks that default page of ERP5 is reachable.
""" """
def _checkERP5IsReachable(self, url): def _checkERP5IsReachable(self, url, verify):
# What happens is that instanciation just create the services, but does not # What happens is that instanciation just create the services, but does not
# wait for ERP5 to be initialized. When this test run ERP5 instance is # wait for ERP5 to be initialized. When this test run ERP5 instance is
# instanciated, but zope is still busy creating the site and haproxy replies # instanciated, but zope is still busy creating the site and haproxy replies
...@@ -51,7 +52,7 @@ class TestPublishedURLIsReachableMixin(object): ...@@ -51,7 +52,7 @@ class TestPublishedURLIsReachableMixin(object):
# erp5 site is not created, with 500 when mysql is not yet reachable, so we # erp5 site is not created, with 500 when mysql is not yet reachable, so we
# retry in a loop until we get a succesful response. # retry in a loop until we get a succesful response.
for i in range(1, 60): for i in range(1, 60):
r = requests.get(url, verify=False) # XXX can we get CA from caucase already ? r = requests.get(url, verify=verify)
if r.status_code != requests.codes.ok: if r.status_code != requests.codes.ok:
delay = i * 2 delay = i * 2
self.logger.warn("ERP5 was not available, sleeping for %ds and retrying", delay) self.logger.warn("ERP5 was not available, sleeping for %ds and retrying", delay)
...@@ -62,19 +63,36 @@ class TestPublishedURLIsReachableMixin(object): ...@@ -62,19 +63,36 @@ class TestPublishedURLIsReachableMixin(object):
self.assertIn("ERP5", r.text) self.assertIn("ERP5", r.text)
def _getCaucaseServiceCACertificate(self):
ca_cert = tempfile.NamedTemporaryFile(
prefix="ca.crt.pem",
mode="w",
delete=False,
)
ca_cert.write(
requests.get(
urlparse.urljoin(
self.getRootPartitionConnectionParameterDict()['caucase-http-url'],
'/cas/crt/ca.crt.pem',
)).text)
self.addCleanup(os.unlink, ca_cert.name)
return ca_cert.name
def test_published_family_default_v6_is_reachable(self): def test_published_family_default_v6_is_reachable(self):
"""Tests the IPv6 URL published by the root partition is reachable. """Tests the IPv6 URL published by the root partition is reachable.
""" """
param_dict = self.getRootPartitionConnectionParameterDict() param_dict = self.getRootPartitionConnectionParameterDict()
self._checkERP5IsReachable( self._checkERP5IsReachable(
urlparse.urljoin(param_dict['family-default-v6'], param_dict['site-id'])) urlparse.urljoin(param_dict['family-default-v6'], param_dict['site-id']),
self._getCaucaseServiceCACertificate())
def test_published_family_default_v4_is_reachable(self): def test_published_family_default_v4_is_reachable(self):
"""Tests the IPv4 URL published by the root partition is reachable. """Tests the IPv4 URL published by the root partition is reachable.
""" """
param_dict = self.getRootPartitionConnectionParameterDict() param_dict = self.getRootPartitionConnectionParameterDict()
self._checkERP5IsReachable( self._checkERP5IsReachable(
urlparse.urljoin(param_dict['family-default'], param_dict['site-id'])) urlparse.urljoin(param_dict['family-default'], param_dict['site-id']),
self._getCaucaseServiceCACertificate())
class TestDefaultParameters(ERP5InstanceTestCase, TestPublishedURLIsReachableMixin): class TestDefaultParameters(ERP5InstanceTestCase, TestPublishedURLIsReachableMixin):
......
...@@ -34,8 +34,10 @@ import unittest ...@@ -34,8 +34,10 @@ import unittest
import urlparse import urlparse
import base64 import base64
import hashlib import hashlib
import logging
import contextlib import contextlib
from BaseHTTPServer import BaseHTTPRequestHandler, HTTPServer from BaseHTTPServer import BaseHTTPRequestHandler
from io import BytesIO from io import BytesIO
import paramiko import paramiko
...@@ -48,30 +50,22 @@ from selenium.webdriver.support import expected_conditions as EC ...@@ -48,30 +50,22 @@ from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import WebDriverWait from selenium.webdriver.support.ui import WebDriverWait
from slapos.testing.testcase import makeModuleSetUpAndTestCaseClass from slapos.testing.testcase import makeModuleSetUpAndTestCaseClass
from slapos.testing.utils import findFreeTCPPort, ImageComparisonTestCase from slapos.testing.utils import findFreeTCPPort, ImageComparisonTestCase, ManagedHTTPServer
setUpModule, SeleniumServerTestCase = makeModuleSetUpAndTestCaseClass( setUpModule, SeleniumServerTestCase = makeModuleSetUpAndTestCaseClass(
os.path.abspath( os.path.abspath(
os.path.join(os.path.dirname(__file__), '..', 'software.cfg'))) os.path.join(os.path.dirname(__file__), '..', 'software.cfg')))
class WebServerMixin(object):
"""Mixin class which provides a simple web server reachable at self.server_url class WebServer(ManagedHTTPServer):
""" class RequestHandler(BaseHTTPRequestHandler):
def setUp(self):
"""Start a minimal web server.
"""
class TestHandler(BaseHTTPRequestHandler):
"""Request handler for our test server. """Request handler for our test server.
The implemented server is: The implemented server is:
- submit q and you'll get a page with q as title - submit q and you'll get a page with q as title
- upload a file and the file content will be displayed in div.uploadedfile - upload a file and the file content will be displayed in div.uploadedfile
""" """
def log_message(self, *args, **kw):
if SeleniumServerTestCase._debug:
BaseHTTPRequestHandler.log_message(self, *args, **kw)
def do_GET(self): def do_GET(self):
self.send_response(200) self.send_response(200)
self.end_headers() self.end_headers()
...@@ -111,18 +105,14 @@ class WebServerMixin(object): ...@@ -111,18 +105,14 @@ class WebServerMixin(object):
</html> </html>
''' % (form['q'].value, file_data)) ''' % (form['q'].value, file_data))
super(WebServerMixin, self).setUp() log_message = logging.getLogger(__name__ + '.WebServer').info
ip = os.environ.get('SLAPOS_TEST_IPV4', '127.0.1.1')
port = findFreeTCPPort(ip)
server = HTTPServer((ip, port), TestHandler)
self.server_process = multiprocessing.Process(target=server.serve_forever)
self.server_process.start()
self.server_url = 'http://%s:%s/' % (ip, port)
def tearDown(self):
self.server_process.terminate() class WebServerMixin(object):
self.server_process.join() """Mixin class which provides a simple web server reachable at self.server_url
super(WebServerMixin, self).tearDown() """
def setUp(self):
self.server_url = self.getManagedResource('web_server', WebServer).url
class BrowserCompatibilityMixin(WebServerMixin): class BrowserCompatibilityMixin(WebServerMixin):
......
...@@ -316,3 +316,4 @@ funcsigs = 1.0.2 ...@@ -316,3 +316,4 @@ funcsigs = 1.0.2
mysqlclient = 1.3.12 mysqlclient = 1.3.12
pexpect = 4.8.0 pexpect = 4.8.0
ptyprocess = 0.6.0 ptyprocess = 0.6.0
typing = 3.7.4.3
...@@ -90,7 +90,7 @@ md5sum = 2f3ddd328ac1c375e483ecb2ef5ffb57 ...@@ -90,7 +90,7 @@ md5sum = 2f3ddd328ac1c375e483ecb2ef5ffb57
[template-balancer] [template-balancer]
filename = instance-balancer.cfg.in filename = instance-balancer.cfg.in
md5sum = bb9a953ce22f7d5188385f0171b6198e md5sum = ecf119142e6b5cd85a2ba397552d2142
[template-haproxy-cfg] [template-haproxy-cfg]
filename = haproxy.cfg.in filename = haproxy.cfg.in
......
...@@ -18,19 +18,52 @@ per partition. No more (undefined result), no less (IndexError). ...@@ -18,19 +18,52 @@ per partition. No more (undefined result), no less (IndexError).
recipe = slapos.recipe.template:jinja2 recipe = slapos.recipe.template:jinja2
mode = 644 mode = 644
[balancer-csr-request-config]
< = jinja2-template-base
template = inline:
[req]
prompt = no
req_extensions = req_ext
distinguished_name = dn
[ dn ]
CN = example.com
[ req_ext ]
subjectAltName = @alt_names
[ alt_names ]
IP.1 = {{ ipv4 }}
{% if ipv6_set -%}
IP.2 = {{ ipv6 }}
{% endif %}
rendered = ${buildout:parts-directory}/${:_buildout_section_name_}/${:_buildout_section_name_}.txt
[balancer-csr-request]
recipe = plone.recipe.command
command = {{ parameter_dict["openssl"] }}/bin/openssl req \
-newkey rsa:2048 \
-batch \
-new \
-nodes \
-keyout '${apache-conf-ssl:key}' \
-config '${balancer-csr-request-config:rendered}' \
-out '${:csr}'
stop-on-error = true
csr = ${directory:etc}/${:_buildout_section_name_}.csr.pem
{{ caucase.updater( {{ caucase.updater(
prefix='caucase-updater', prefix='caucase-updater',
buildout_bin_directory=parameter_dict['bin-directory'], buildout_bin_directory=parameter_dict['bin-directory'],
updater_path='${directory:services-on-watch}/caucase-updater', updater_path='${directory:services-on-watch}/caucase-updater',
url=ssl_parameter_dict['caucase-url'], url=ssl_parameter_dict['caucase-url'],
data_dir='${directory:srv}/caucase-updater', data_dir='${directory:srv}/caucase-updater',
crt_path='${apache-conf-ssl:caucase-cert}', crt_path='${apache-conf-ssl:cert}',
ca_path='${directory:srv}/caucase-updater/ca.crt', ca_path='${directory:srv}/caucase-updater/ca.crt',
crl_path='${directory:srv}/caucase-updater/crl.pem', crl_path='${directory:srv}/caucase-updater/crl.pem',
key_path='${apache-conf-ssl:caucase-key}', key_path='${apache-conf-ssl:key}',
on_renew='${apache-graceful:output}', on_renew='${apache-graceful:output}',
max_sleep=ssl_parameter_dict.get('max-crl-update-delay', 1.0), max_sleep=ssl_parameter_dict.get('max-crl-update-delay', 1.0),
template_csr_pem=ssl_parameter_dict.get('csr'), template_csr_pem=ssl_parameter_dict.get('csr'),
template_csr=None if ssl_parameter_dict.get('csr') else '${balancer-csr-request:csr}',
openssl=parameter_dict['openssl'] ~ '/bin/openssl', openssl=parameter_dict['openssl'] ~ '/bin/openssl',
)}} )}}
{% do section('caucase-updater') -%} {% do section('caucase-updater') -%}
...@@ -176,9 +209,6 @@ hash-files = ${haproxy-cfg:rendered} ...@@ -176,9 +209,6 @@ hash-files = ${haproxy-cfg:rendered}
[apache-conf-ssl] [apache-conf-ssl]
cert = ${directory:apache-conf}/apache.crt cert = ${directory:apache-conf}/apache.crt
key = ${directory:apache-conf}/apache.pem key = ${directory:apache-conf}/apache.pem
# XXX caucase certificate is not supported by caddy for now
caucase-cert = ${directory:apache-conf}/apache-caucase.crt
caucase-key = ${directory:apache-conf}/apache-caucase.pem
{% if frontend_caucase_url_list -%} {% if frontend_caucase_url_list -%}
depends = ${caucase-updater-housekeeper-run:recipe} depends = ${caucase-updater-housekeeper-run:recipe}
ca-cert-dir = ${directory:apache-ca-cert-dir} ca-cert-dir = ${directory:apache-ca-cert-dir}
...@@ -201,19 +231,6 @@ context = key content {{content_section_name}}:content ...@@ -201,19 +231,6 @@ context = key content {{content_section_name}}:content
mode = {{ mode }} mode = {{ mode }}
{%- endmacro %} {%- endmacro %}
[apache-ssl]
{% if ssl_parameter_dict.get('key') -%}
key = ${apache-ssl-key:rendered}
cert = ${apache-ssl-cert:rendered}
{{ simplefile('apache-ssl-key', '${apache-conf-ssl:key}', ssl_parameter_dict['key']) }}
{{ simplefile('apache-ssl-cert', '${apache-conf-ssl:cert}', ssl_parameter_dict['cert']) }}
{% else %}
recipe = plone.recipe.command
command = "{{ parameter_dict['openssl'] }}/bin/openssl" req -newkey rsa -batch -new -x509 -days 3650 -nodes -keyout "${:key}" -out "${:cert}"
key = ${apache-conf-ssl:key}
cert = ${apache-conf-ssl:cert}
{%- endif %}
[apache-conf-parameter-dict] [apache-conf-parameter-dict]
backend-list = {{ dumps(apache_dict.values()) }} backend-list = {{ dumps(apache_dict.values()) }}
zope-virtualhost-monster-backend-dict = {{ dumps(zope_virtualhost_monster_backend_dict) }} zope-virtualhost-monster-backend-dict = {{ dumps(zope_virtualhost_monster_backend_dict) }}
...@@ -225,8 +242,8 @@ access-log = ${directory:log}/apache-access.log ...@@ -225,8 +242,8 @@ access-log = ${directory:log}/apache-access.log
# Apache 2.4's default value (60 seconds) can be a bit too short # Apache 2.4's default value (60 seconds) can be a bit too short
timeout = 300 timeout = 300
# Basic SSL server configuration # Basic SSL server configuration
cert = ${apache-ssl:cert} cert = ${apache-conf-ssl:cert}
key = ${apache-ssl:key} key = ${apache-conf-ssl:key}
cipher = cipher =
ssl-session-cache = ${directory:log}/apache-ssl-session-cache ssl-session-cache = ${directory:log}/apache-ssl-session-cache
{% if frontend_caucase_url_list -%} {% if frontend_caucase_url_list -%}
......
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment