Commit 3ef32f24 authored by Lu Xu's avatar Lu Xu 👀

Add end-to-end-testing SR

See merge request !1486
parents 0bfb7e5f ccd7619b
# End To End Testing
This software release is used to run end-to-end test of SlapOS softwares on an actual SlapOS cloud such as Rapid.Space. Since it can supply softwares and request instances on an actual cloud, it needs a SlaPOS client certificate and the URLs of your tests.
## Input parameters
```
{
"client.crt": <content of client.crt>,
"client.key": <content of client.key>,
"master-url": <url of SlapOS master>,
"tests": [
{
"url": "<url of test1 script>",
"md5sum": "MD5sum of test1 script"
},
{
"url": "<url of test2 python script>",
"md5sum": "MD5sum of test2 python script"
},
...
]
}
```
Example:
(`e2e-parameters.json`)
```
{
"client.crt": "Certificate:...-----END CERTIFICATE-----\n",
"master-url": "https://slap.vifib.com",
"tests": [
{
"url": "https://lab.nexedi.com/lu.xu/slapos/raw/feat/end-to-end-testing/software/end-to-end-testing/test_test.py",
"md5sum": "c074373dbb4154aa924ef5781dade7a0"
}
]
}
```
## Generate client certificate
Follow [How To Set Up SlapOS Client](https://handbook.rapid.space/user/rapidspace-HowTo.Setup.SlapOS.Client) to prepare `slapos-client.cfg` if you don't have one.
A convenience script `generate_parameters.py` is provided to compute these parameters in JSON format from an existing SlapOS client configuration:
```
python3 generate_parameters.py --cfg <absolute path to slapos-client.cfg> -o <output path>
```
## Adding tests
There are 3 example tests available in end-to-end testing SR:
- test_test.py
Simple successful test and failed test
- test_kvm.py
Request a KVM instance with published SR and verify one of the connection parameters
- test_health.py
Request a Monitor instance with published SR and log promises output
All tests should be written in Python with a `.py` extension and should have names that start with `test_`.
Once your test is prepared, you have the option to input a URL and its corresponding md5sum as parameters. This will enable the end-to-end testing instance to automatically detect the test.
## Running tests
### In Nexedi ERP5 automated test environment (testnodes)
When performing tests on a software release (SR) using the ERR5 test suite, you can use "SlapOS.SoftwareReleases.IntegrationTest". Additionally, make sure to fill in the "Slapos Parameters" field with the content of the input parameter mentioned above.
### In Theia locally
#### Using virtualenv
Using slapos.core for quick testing:
```
python3 -m venv testenv
source testenv/bin/activate
pip install -e path/to/my/slapos.core
export SLAPOS_E2E_TEST_CLIENT_CFG=my_test_client_cfg
export SLAPOS_E2E_TEST_LOG_FILE=my_test_log_file
python -m unittest my_test_file_in_development
# edit e2e.py in path/to/my/slapos.core or the instanciated one if improvements are needed
```
#### Using an instance of software/end-to-end-testing
1. Setup and instantiate the runner
```
slapos supply ~/srv/project/slapos/software/end-to-end-testing/software.cfg slaprunner
slapos request <e2e_instance_name> ~/srv/project/slapos/software/end-to-end-testing/software.cfg --parameters-file <e2e_parameter_json_file>
```
Your tests should be listed in <e2e_parameter_json_file> with URL and MD5sum
2. Go to instance directory and run test
```
cd ~/srv/runner/instance/slappartX
./bin/runTestSuite
```
Downloaded tests and the reusable `e2e.py` script can be found in the `~/srv/runner/instance/slappartX/var/tests/` directory.
To quickly test, you have the option to modify the test script directly here(`~/srv/runner/instance/slappartX/var/tests/`). After making the necessary changes, you can relaunch the tests by running `./bin/runTestSuite`.
## FAQ
Q1. What is the difference between `slapos-sr-testing` and `end-to-end-testing`?
- slapos-sr-testing requests SRs on a slapproxy in an SlapOSStandalone inside the slapos-sr-testing instance (same kind of thing as Theia or webrunner).
- end-to-end-testing requests SRs on the actual master, in real compute nodes (COMP-XXX). To do this it needs a slapos certificate which is passed as instance parameter to end-to-end-testing instance in the test suite.
So unlike slapos-sr-testing, end-to-end-testing does not contain the SRs it tests, it merely runs the python tests scripts (like mentioned `test_kvm.py`) and integrates with the ERP5 test suite. This also means we cannot access the files in the partition of the tested SRs, as those are on other computers. All we have access to is what a normal user requesting on panel would have access to.
[instance.cfg]
filename = instance.cfg.in
md5sum = 962830010e0a257d52c22141db3d34cf
import argparse
import configparser
import json
def main():
parser = argparse.ArgumentParser()
parser.add_argument('--cfg', required=True)
parser.add_argument('-o', '--output', required=True)
args = parser.parse_args()
configp = configparser.ConfigParser()
configp.read(args.cfg)
with open(configp.get('slapconsole', 'cert_file')) as f:
crt = f.read()
with open(configp.get('slapconsole', 'key_file')) as f:
key = f.read()
url = configp.get('slapos', 'master_url')
with open(args.output, 'w') as f:
json.dump(
{
'client.crt': crt,
'client.key': key,
'master-url': url
}, f, indent=2)
if __name__ == '__main__':
main()
{
"$schema": "http://json-schema.org/draft-07/schema#",
"type": "object",
"properties": {
"client.crt": {
"type": "string",
"pattern": "^Certificate:.+\\n-----END CERTIFICATE-----\\n$",
"default": "Certificate:\ndefault-client-crt\n-----END CERTIFICATE-----\n"
},
"client.key": {
"type": "string",
"pattern": "^-----BEGIN PRIVATE KEY-----.+\\n-----END PRIVATE KEY-----\\n$",
"default": "-----BEGIN PRIVATE KEY-----\ndefault-client-key\n-----END PRIVATE KEY-----\n"
},
"master-url": {
"type": "string",
"format": "uri",
"default": "https://slap.vifib.com"
},
"tests": {
"type": "array",
"items": {
"type": "object",
"properties": {
"url": {
"type": "string",
"format": "uri"
},
"md5sum": {
"type": "string",
"pattern": "^[a-f0-9]{32}$"
}
},
"additionalProperties": false
}
}
},
"additionalProperties": false
}
{
"$schema": "http://json-schema.org/draft-07/schema#",
"description": "Values returned by End to End Testing instanciation",
"additionalProperties": false,
"properties": {},
"type": "object"
}
[buildout]
eggs-directory = ${buildout:eggs-directory}
develop-eggs-directory = ${buildout:develop-eggs-directory}
extends =
${nxdtest-instance.cfg:output}
parts =
.nxdtest
[slap-configuration]
recipe = slapos.cookbook:slapconfiguration.serialised
computer = $${slap-connection:computer-id}
partition = $${slap-connection:partition-id}
url = $${slap-connection:server-url}
key = $${slap-connection:key-file}
cert = $${slap-connection:cert-file}
[directory]
recipe = slapos.cookbook:mkdirectory
home = $${buildout:directory}
bin = $${buildout:directory}/bin
etc = $${buildout:directory}/etc
var = $${buildout:directory}/var
cfg = $${buildout:directory}/.slapos
nxdtestdir = $${:var}/nxdtest
log = $${:var}/log
[client.crt]
recipe = slapos.recipe.template
output = $${directory:cfg}/client.crt
inline = $${slap-configuration:configuration.client.crt}
[client.key]
recipe = slapos.recipe.template
output = $${directory:cfg}/client.key
inline = $${slap-configuration:configuration.client.key}
[slapos-client.cfg]
recipe = slapos.recipe.template
output = $${directory:cfg}/slapos-client.cfg
inline =
[slapos]
master_url = $${slap-configuration:configuration.master-url}
[slapconsole]
cert_file = $${client.crt:output}
key_file = $${client.key:output}
[env.sh]
recipe = slapos.recipe.template:jinja2
output = $${directory:cfg}/env.sh
inline =
export HOME=$${directory:home}
[runTestSuite]
# extended from stack/nxdtest
env.sh = $${env.sh:output}
workdir = $${directory:nxdtestdir}
[.nxdtest]
recipe = slapos.recipe.template:jinja2
output = $${runTestSuite:workdir}/.nxdtest
python_for_test = ${python_for_test:executable}
testdir = $${tests:location}
log-output = $${directory:log}/e2e-testing.log
context =
key python_for_test :python_for_test
key testdir :testdir
key slapos_cfg slapos-client.cfg:output
key log_file :log-output
inline =
import os
directory = "{{ testdir }}"
slapos_cfg = {{ repr(slapos_cfg) }}
log_file = {{ repr(log_file) }}
dir_list = os.listdir(directory)
for filename in dir_list:
name, ext = os.path.splitext(filename)
if name.startswith('test') and ext == '.py':
TestCase(
name,
[{{ repr(python_for_test) }} , '-m', 'unittest', '-v', name],
cwd=directory,
env={'SLAPOS_E2E_TEST_CLIENT_CFG': slapos_cfg,
'SLAPOS_E2E_TEST_LOG_FILE': log_file},
summaryf=UnitTest.summary,
)
[tests]
recipe = slapos.recipe.build
tests = $${slap-configuration:configuration.tests}
location = $${directory:var}/tests
install =
import os
os.mkdir(location)
buildout_offline = self.buildout['buildout']['offline']
try:
# Allow to do self.download() which can only be used in "online" mode
self.buildout['buildout']['offline'] = 'false'
for i, test in enumerate(options['tests']):
tmp = self.download(test['url'], test['md5sum'])
path = os.path.join(location, 'test%s.py' % i)
os.rename(tmp, path)
finally:
# reset the parameter
self.buildout['buildout']['offline'] = buildout_offline
[buildout]
extends =
../../component/pygolang/buildout.cfg
../../stack/slapos.cfg
../../stack/nxdtest.cfg
buildout.hash.cfg
parts =
instance.cfg
slapos-cookbook
[instance.cfg]
recipe = slapos.recipe.template
output = ${buildout:directory}/instance.cfg
url = ${:_profile_base_location_}/${:filename}
[e2e.py]
recipe = slapos.recipe.build:download
output = ${buildout:directory}/${:filename}
url = ${:_profile_base_location_}/${:filename}
[python_for_test]
<= python-interpreter
interpreter = python_for_test
executable = ${buildout:bin-directory}/${:interpreter}
depends = ${lxml-python:egg}
eggs =
${pygolang:egg}
slapos.core
websocket-client
requests
[versions]
websocket-client = 1.4.2
{
"name": "End-To-End-Testing",
"description": "End-To-End Testing on SlapOS Cloud",
"serialisation": "json-in-xml",
"software-type": {
"default": {
"title": "default",
"software-type": "default",
"description": "default",
"request": "instance-input-schema.json",
"response": "instance-output-schema.json"
}
}
}
import slapos.testing.e2e as e2e
import time
class HealthTest(e2e.EndToEndTestCase):
@classmethod
def test_health_promise_feed(self):
instance_name = e2e.time.strftime('e2e-test-health-%Y-%B-%d-%H:%M:%S')
product = self.product.slapmonitor
parameter_dict = {}
self.request(
self.product.slapmonitor,
instance_name,
software_type='default',
filter_kw={"computer_guid": "COMP-4057"})
self.waitUntilGreen(instance_name)
self.connection_dict = self.getInstanceInfos(instance_name).connection_dict
resp, url = self.waitUntilMonitorURLReady(instance_name=instance_name)
self.getMonitorPromises(resp.content)
import slapos.testing.e2e as e2e
class KvmTest(e2e.EndToEndTestCase):
def test(self):
instance_name = e2e.time.strftime('e2e-test-kvm-%Y-%B-%d-%H:%M:%S')
# instance_name = 'e2e-kvm-test' # avoid timestamp to reuse instance
self.request(self.product.kvm, instance_name)
self.waitUntilGreen(instance_name)
connection_dict = self.request(self.product.kvm, instance_name)
self.assertIn('url', connection_dict)
import unittest
class Test(unittest.TestCase):
def test_fail(self):
self.assertEqual(0, 1)
def test_succeed(self):
self.assertEqual(0, 0)
......@@ -303,7 +303,7 @@ simplegeneric = 0.8.1
singledispatch = 3.4.0.3
six = 1.16.0
slapos.cookbook = 1.0.329
slapos.core = 1.10.7
slapos.core = 1.10.8
slapos.extension.shared = 1.0
slapos.libnetworkcache = 0.25
slapos.rebootstrap = 4.5
......
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment