ERP5: Test balancer partition and use caucase certificate for balancer

Revert f8f72a17 ([erp5] don't use caucase generated certificate for now, 2019-03-12) since nothing prevents us drom using caucase certificate now.
 
Use [managed resources](nexedi/slapos.core!259) to simplify existing tests and introduce tests for:

## Access Log

 - [x] balancer partition should produce logs in apache "combined" log format with microsecond timing of requests.
 - [x] these logs should be rotated daily
 - [x] an [apachedex](https://lab.nexedi.com/nexedi/apachedex) report is ran on these logs daily.

## Balancing

 - [x] requests are balanced to multiple backends using round-robin algorithm
 - [x] if backend is down it is excluded
 - [x] a "sticky cookie" is used so that clients are associated to the same backend
    - [x] the cookie is set by balancer
    - [x] when client comes with a cookie it "sticks" on the associated backend
    - [x] if "sticked" backend is down, another backend will be used

## Content-Encoding

 - [x] balancer encodes responses in gzip for some configured content types.

## HTTP

 - [x] Server uses HTTP/1.1 or more and keep connection with clients

## TLS (server certificate)

In this MR we also change apache to use a caucase managed certificate and add test coverage for:

 - [x] balancer listen on https with a certificate that can be verified using the CA from caucase.
 - [x] balancer uses the new certificate when its own certificate is renewed.

But we don't add support for:
 -  ~~balancer can be instantiated with a certificate and key passed as SlapOS request parameters (code [here](https://lab.nexedi.com/nexedi/slapos/blob/757c1a4ddee93659d5e2649e4252d87bf9494566/stack/erp5/instance-balancer.cfg.in#L208-213))~~ this use case is the job of caucase, so we no longer support this.

## TLS (client certificate)
 - [x] balancer verifies frontend certificates from frontend caucases ( also tested in "Forwarded-For" section )
 - [x] if frontend provided a verified certificate, balancer set `remote-user` header
 - [x] balancer updates CRL from caucases ( `caucase-updater-housekeeper` )
 - (NOT TESTED) balancer updates CA certificate from caucase ( `caucase-updater-housekeeper` ). Since this is would be complex to test and basic functionality of `caucase-updater-housekeeper` for frontend caucases is covered by CRL test, we don't test this for simplicity.

## "Forwarded-For" header

This was also covered by existing tests:  

 - [x] balancer set `X-Forwarded-For` header when frontend certificate can be verified
 - [x] balancer strips existing `X-Forwarded-For`

## Integration with the rest of ERP5 software release

This was also covered by existing tests:  

- [x] The https URL of each Zope family is published and replies properly
- [x] Some https URLs are generated for `runUnitTest`, so that test run with an https certificate. This is also covered by regular ERP5 functional tests.

See merge request nexedi/slapos!840

software/erp5/test/setup.py

@@ -33,7 +33,7 @@ with open("README.md") as f:
 
 setup(name=name,
       version=version,
-      description="Test for SlapOS' ERP5 software releae",
+      description="Test for SlapOS' ERP5 software release",
       long_description=long_description,
       long_description_content_type='text/markdown',
       maintainer="Nexedi",
@@ -50,8 +50,9 @@ setup(name=name,
         'mysqlclient',
         'backports.lzma',
         'cryptography',
+        'pexpect',
         'pyOpenSSL',
-        ],
-      zip_safe=True,
+        'typing; python_version<"3"',
+      ],
       test_suite='test',
     )

software/erp5/test/test/test_erp5.py

@@ -31,6 +31,7 @@ import glob
 import urlparse
 import socket
 import time
+import tempfile
 
 import psutil
 import requests
@@ -43,7 +44,7 @@ setUpModule # pyflakes
 class TestPublishedURLIsReachableMixin(object):
   """Mixin that checks that default page of ERP5 is reachable.
   """
-  def _checkERP5IsReachable(self, url):
+  def _checkERP5IsReachable(self, url, verify):
     # What happens is that instanciation just create the services, but does not
     # wait for ERP5 to be initialized. When this test run ERP5 instance is
     # instanciated, but zope is still busy creating the site and haproxy replies
@@ -51,7 +52,7 @@ class TestPublishedURLIsReachableMixin(object):
     # erp5 site is not created, with 500 when mysql is not yet reachable, so we
     # retry in a loop until we get a succesful response.
     for i in range(1, 60):
-      r = requests.get(url, verify=False)  # XXX can we get CA from caucase already ?
+      r = requests.get(url, verify=verify)
       if r.status_code != requests.codes.ok:
         delay = i * 2
         self.logger.warn("ERP5 was not available, sleeping for %ds and retrying", delay)
@@ -62,19 +63,36 @@ class TestPublishedURLIsReachableMixin(object):
 
     self.assertIn("ERP5", r.text)
 
+  def _getCaucaseServiceCACertificate(self):
+    ca_cert = tempfile.NamedTemporaryFile(
+        prefix="ca.crt.pem",
+        mode="w",
+        delete=False,
+    )
+    ca_cert.write(
+        requests.get(
+            urlparse.urljoin(
+                self.getRootPartitionConnectionParameterDict()['caucase-http-url'],
+                '/cas/crt/ca.crt.pem',
+            )).text)
+    self.addCleanup(os.unlink, ca_cert.name)
+    return ca_cert.name
+
   def test_published_family_default_v6_is_reachable(self):
     """Tests the IPv6 URL published by the root partition is reachable.
     """
     param_dict = self.getRootPartitionConnectionParameterDict()
     self._checkERP5IsReachable(
-      urlparse.urljoin(param_dict['family-default-v6'], param_dict['site-id']))
+      urlparse.urljoin(param_dict['family-default-v6'], param_dict['site-id']),
+      self._getCaucaseServiceCACertificate())
 
   def test_published_family_default_v4_is_reachable(self):
     """Tests the IPv4 URL published by the root partition is reachable.
     """
     param_dict = self.getRootPartitionConnectionParameterDict()
     self._checkERP5IsReachable(
-      urlparse.urljoin(param_dict['family-default'], param_dict['site-id']))
+      urlparse.urljoin(param_dict['family-default'], param_dict['site-id']),
+      self._getCaucaseServiceCACertificate())
 
 
 class TestDefaultParameters(ERP5InstanceTestCase, TestPublishedURLIsReachableMixin):

software/seleniumserver/test/test.py

@@ -34,8 +34,10 @@ import unittest
 import urlparse
 import base64
 import hashlib
+import logging
 import contextlib
-from BaseHTTPServer import BaseHTTPRequestHandler, HTTPServer
+from BaseHTTPServer import BaseHTTPRequestHandler
+
 from io import BytesIO
 
 import paramiko
@@ -48,81 +50,69 @@ from selenium.webdriver.support import expected_conditions as EC
 from selenium.webdriver.support.ui import WebDriverWait
 
 from slapos.testing.testcase import makeModuleSetUpAndTestCaseClass
-from slapos.testing.utils import findFreeTCPPort, ImageComparisonTestCase
+from slapos.testing.utils import findFreeTCPPort, ImageComparisonTestCase, ManagedHTTPServer
 
 setUpModule, SeleniumServerTestCase = makeModuleSetUpAndTestCaseClass(
     os.path.abspath(
         os.path.join(os.path.dirname(__file__), '..', 'software.cfg')))
 
 
+
+class WebServer(ManagedHTTPServer):
+  class RequestHandler(BaseHTTPRequestHandler):
+    """Request handler for our test server.
+
+    The implemented server is:
+      - submit q and you'll get a page with q as title
+      - upload a file and the file content will be displayed in div.uploadedfile
+    """
+    def do_GET(self):
+      self.send_response(200)
+      self.end_headers()
+      self.wfile.write(
+          '''
+        <html>
+          <title>Test page</title>
+          <body>
+            <style> p { font-family: Arial; } </style>
+            <form action="/" method="POST" enctype="multipart/form-data">
+              <input name="q" type="text"></input>
+              <input name="f" type="file" ></input>
+              <input type="submit" value="I'm feeling lucky"></input>
+            </form>
+            <p>the quick brown fox jumps over the lazy dog</p>
+          </body>
+        </html>''')
+
+    def do_POST(self):
+      form = cgi.FieldStorage(
+          fp=self.rfile,
+          headers=self.headers,
+          environ={
+              'REQUEST_METHOD': 'POST',
+              'CONTENT_TYPE': self.headers['Content-Type'],
+          })
+      self.send_response(200)
+      self.end_headers()
+      file_data = 'no file'
+      if form.has_key('f'):
+        file_data = form['f'].file.read()
+      self.wfile.write(
+          '''
+        <html>
+          <title>%s</title>
+          <div>%s</div>
+        </html>
+      ''' % (form['q'].value, file_data))
+
+    log_message = logging.getLogger(__name__ + '.WebServer').info
+
+
 class WebServerMixin(object):
   """Mixin class which provides a simple web server reachable at self.server_url
   """
   def setUp(self):
-    """Start a minimal web server.
-    """
-    class TestHandler(BaseHTTPRequestHandler):
-      """Request handler for our test server.
-
-      The implemented server is:
-       - submit q and you'll get a page with q as title
-       - upload a file and the file content will be displayed in div.uploadedfile
-      """
-      def log_message(self, *args, **kw):
-        if SeleniumServerTestCase._debug:
-          BaseHTTPRequestHandler.log_message(self, *args, **kw)
-
-      def do_GET(self):
-        self.send_response(200)
-        self.end_headers()
-        self.wfile.write(
-            '''
-          <html>
-            <title>Test page</title>
-            <body>
-              <style> p { font-family: Arial; } </style>
-              <form action="/" method="POST" enctype="multipart/form-data">
-                <input name="q" type="text"></input>
-                <input name="f" type="file" ></input>
-                <input type="submit" value="I'm feeling lucky"></input>
-              </form>
-              <p>the quick brown fox jumps over the lazy dog</p>
-            </body>
-          </html>''')
-
-      def do_POST(self):
-        form = cgi.FieldStorage(
-            fp=self.rfile,
-            headers=self.headers,
-            environ={
-                'REQUEST_METHOD': 'POST',
-                'CONTENT_TYPE': self.headers['Content-Type'],
-            })
-        self.send_response(200)
-        self.end_headers()
-        file_data = 'no file'
-        if form.has_key('f'):
-          file_data = form['f'].file.read()
-        self.wfile.write(
-            '''
-          <html>
-            <title>%s</title>
-            <div>%s</div>
-          </html>
-        ''' % (form['q'].value, file_data))
-
-    super(WebServerMixin, self).setUp()
-    ip = os.environ.get('SLAPOS_TEST_IPV4', '127.0.1.1')
-    port = findFreeTCPPort(ip)
-    server = HTTPServer((ip, port), TestHandler)
-    self.server_process = multiprocessing.Process(target=server.serve_forever)
-    self.server_process.start()
-    self.server_url = 'http://%s:%s/' % (ip, port)
-
-  def tearDown(self):
-    self.server_process.terminate()
-    self.server_process.join()
-    super(WebServerMixin, self).tearDown()
+    self.server_url = self.getManagedResource('web_server', WebServer).url
 
 
 class BrowserCompatibilityMixin(WebServerMixin):

software/slapos-sr-testing/software.cfg

@@ -316,3 +316,4 @@ funcsigs = 1.0.2
 mysqlclient = 1.3.12
 pexpect = 4.8.0
 ptyprocess = 0.6.0
+typing = 3.7.4.3

stack/erp5/buildout.hash.cfg

@@ -90,7 +90,7 @@ md5sum = 2f3ddd328ac1c375e483ecb2ef5ffb57
 
 [template-balancer]
 filename = instance-balancer.cfg.in
-md5sum = bb9a953ce22f7d5188385f0171b6198e
+md5sum = ecf119142e6b5cd85a2ba397552d2142
 
 [template-haproxy-cfg]
 filename = haproxy.cfg.in

stack/erp5/instance-balancer.cfg.in

@@ -18,19 +18,52 @@ per partition. No more (undefined result), no less (IndexError).
 recipe = slapos.recipe.template:jinja2
 mode = 644
 
+[balancer-csr-request-config]
+< = jinja2-template-base
+template = inline:
+    [req]
+    prompt = no
+    req_extensions = req_ext
+    distinguished_name = dn
+    [ dn ]
+    CN = example.com
+    [ req_ext ]
+    subjectAltName = @alt_names
+    [ alt_names ]
+    IP.1 =  {{ ipv4 }}
+    {% if ipv6_set -%}
+    IP.2 =  {{ ipv6 }}
+    {% endif %}
+rendered = ${buildout:parts-directory}/${:_buildout_section_name_}/${:_buildout_section_name_}.txt
+
+[balancer-csr-request]
+recipe = plone.recipe.command
+command = {{ parameter_dict["openssl"] }}/bin/openssl req \
+  -newkey rsa:2048 \
+  -batch \
+  -new \
+  -nodes \
+  -keyout '${apache-conf-ssl:key}' \
+  -config '${balancer-csr-request-config:rendered}' \
+  -out '${:csr}'
+stop-on-error = true
+csr = ${directory:etc}/${:_buildout_section_name_}.csr.pem
+
+
 {{ caucase.updater(
      prefix='caucase-updater',
      buildout_bin_directory=parameter_dict['bin-directory'],
      updater_path='${directory:services-on-watch}/caucase-updater',
      url=ssl_parameter_dict['caucase-url'],
      data_dir='${directory:srv}/caucase-updater',
-     crt_path='${apache-conf-ssl:caucase-cert}',
+     crt_path='${apache-conf-ssl:cert}',
      ca_path='${directory:srv}/caucase-updater/ca.crt',
      crl_path='${directory:srv}/caucase-updater/crl.pem',
-     key_path='${apache-conf-ssl:caucase-key}',
+     key_path='${apache-conf-ssl:key}',
      on_renew='${apache-graceful:output}',
      max_sleep=ssl_parameter_dict.get('max-crl-update-delay', 1.0),
      template_csr_pem=ssl_parameter_dict.get('csr'),
+     template_csr=None if ssl_parameter_dict.get('csr') else '${balancer-csr-request:csr}',
      openssl=parameter_dict['openssl'] ~ '/bin/openssl',
 )}}
 {% do section('caucase-updater') -%}
@@ -176,9 +209,6 @@ hash-files = ${haproxy-cfg:rendered}
 [apache-conf-ssl]
 cert = ${directory:apache-conf}/apache.crt
 key = ${directory:apache-conf}/apache.pem
-# XXX caucase certificate is not supported by caddy for now
-caucase-cert = ${directory:apache-conf}/apache-caucase.crt
-caucase-key = ${directory:apache-conf}/apache-caucase.pem
 {% if frontend_caucase_url_list -%}
 depends = ${caucase-updater-housekeeper-run:recipe}
 ca-cert-dir = ${directory:apache-ca-cert-dir}
@@ -201,19 +231,6 @@ context = key content {{content_section_name}}:content
 mode = {{ mode }}
 {%- endmacro %}
 
-[apache-ssl]
-{% if ssl_parameter_dict.get('key') -%}
-key = ${apache-ssl-key:rendered}
-cert = ${apache-ssl-cert:rendered}
-{{ simplefile('apache-ssl-key', '${apache-conf-ssl:key}', ssl_parameter_dict['key']) }}
-{{ simplefile('apache-ssl-cert', '${apache-conf-ssl:cert}', ssl_parameter_dict['cert']) }}
-{% else %}
-recipe = plone.recipe.command
-command = "{{ parameter_dict['openssl'] }}/bin/openssl" req -newkey rsa -batch -new -x509 -days 3650 -nodes -keyout "${:key}" -out "${:cert}"
-key = ${apache-conf-ssl:key}
-cert = ${apache-conf-ssl:cert}
-{%- endif %}
-
 [apache-conf-parameter-dict]
 backend-list = {{ dumps(apache_dict.values()) }}
 zope-virtualhost-monster-backend-dict = {{ dumps(zope_virtualhost_monster_backend_dict) }}
@@ -225,8 +242,8 @@ access-log = ${directory:log}/apache-access.log
 # Apache 2.4's default value (60 seconds) can be a bit too short
 timeout = 300
 # Basic SSL server configuration
-cert = ${apache-ssl:cert}
-key = ${apache-ssl:key}
+cert = ${apache-conf-ssl:cert}
+key = ${apache-conf-ssl:key}
 cipher =
 ssl-session-cache = ${directory:log}/apache-ssl-session-cache
 {% if frontend_caucase_url_list -%}