Commit 4343f31d authored by Łukasz Nowak's avatar Łukasz Nowak

feature/kvm: User selectable list of boot ISOs

See merge request nexedi/slapos!807
parents ecc880d1 92130619
...@@ -18,9 +18,8 @@ KVM instance (1GB of RAM, 10GB of SSD, one core) ...@@ -18,9 +18,8 @@ KVM instance (1GB of RAM, 10GB of SSD, one core)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Note that the KVM instance will try to request a frontend slave instance in order Note that the KVM instance will try to request a frontend slave instance in order
to be accessible from IPv4. to be accessible from IPv4::
::
myawesomekvm = request( myawesomekvm = request(
software_release=kvm, software_release=kvm,
partition_reference="My awesome KVM", partition_reference="My awesome KVM",
...@@ -97,11 +96,10 @@ KVM instance parameters: ...@@ -97,11 +96,10 @@ KVM instance parameters:
'th', 'tr'] 'th', 'tr']
Resilient KVM instance Resilient KVM instance
~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~
Like KVM instance, but backed-up (with history) in two places. Like KVM instance, but backed-up (with history) in two places::
::
kvm = 'https://lab.nexedi.com/nexedi/slapos/raw/slapos-0.188/software/kvm/software.cfg' kvm = 'https://lab.nexedi.com/nexedi/slapos/raw/slapos-0.188/software/kvm/software.cfg'
myresilientkvm = request( myresilientkvm = request(
software_release=kvm, software_release=kvm,
...@@ -119,3 +117,18 @@ files for more instance parameters (cpu-count, ram-size, disk-size, specific loc ...@@ -119,3 +117,18 @@ files for more instance parameters (cpu-count, ram-size, disk-size, specific loc
Then, if you want one of the two clones to takeover, you need to login into Then, if you want one of the two clones to takeover, you need to login into
the hosting machine, go to the partition of the clone, and invoke bin/takeover. the hosting machine, go to the partition of the clone, and invoke bin/takeover.
Technical notes
---------------
Updating boot-image-url-select
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
* download the new OS installation image
* calculate it's sha512sum and store as <SHA512>
* calculate it's md5sum and store as <MD5>
* upload it to shacache
* construct download url: ``https://shacache.nxdcdn.com/<SHA512>#<MD5>``
* update the ``boot-image-url-select`` in:
* ``instance-kvm-input-schema.json``
* ``instance-kvm-cluster-input-schema.json``
...@@ -19,11 +19,11 @@ md5sum = e6d5c7bb627b4f1d3e7c99721b7c58fe ...@@ -19,11 +19,11 @@ md5sum = e6d5c7bb627b4f1d3e7c99721b7c58fe
[template-kvm] [template-kvm]
filename = instance-kvm.cfg.jinja2 filename = instance-kvm.cfg.jinja2
md5sum = 50e78a2a34efe09afab161ae1c1efd46 md5sum = 23493c541efef97ac5fe435114910b8e
[template-kvm-cluster] [template-kvm-cluster]
filename = instance-kvm-cluster.cfg.jinja2.in filename = instance-kvm-cluster.cfg.jinja2.in
md5sum = a4788112008cd0b38a57cd28f7252fbd md5sum = bdf8549a76ec61e125d51a05e611e004
[template-kvm-resilient] [template-kvm-resilient]
filename = instance-kvm-resilient.cfg.jinja2 filename = instance-kvm-resilient.cfg.jinja2
...@@ -55,7 +55,7 @@ md5sum = b7e87479a289f472b634a046b44b5257 ...@@ -55,7 +55,7 @@ md5sum = b7e87479a289f472b634a046b44b5257
[template-kvm-run] [template-kvm-run]
filename = template/template-kvm-run.in filename = template/template-kvm-run.in
md5sum = dd1f581f34cf5a0b627576771347c710 md5sum = 4a6f149177a453a13436f320f6841518
[template-kvm-controller] [template-kvm-controller]
filename = template/kvm-controller-run.in filename = template/kvm-controller-run.in
...@@ -87,4 +87,4 @@ md5sum = 7e4b54f8172c364bd12d28b07f8b1600 ...@@ -87,4 +87,4 @@ md5sum = 7e4b54f8172c364bd12d28b07f8b1600
[image-download-config-creator] [image-download-config-creator]
_update_hash_filename_ = template/image-download-config-creator.py _update_hash_filename_ = template/image-download-config-creator.py
md5sum = 7f6cd75096695922fddbba1c9292cef5 md5sum = 54261e418ab9860efe73efd514c4d47f
...@@ -42,12 +42,6 @@ ...@@ -42,12 +42,6 @@
"type": "string", "type": "string",
"format": "uri", "format": "uri",
"default": "http://git.erp5.org/gitweb/slapos.git/blob_plain/HEAD:/software/apache-frontend/software.cfg" "default": "http://git.erp5.org/gitweb/slapos.git/blob_plain/HEAD:/software/apache-frontend/software.cfg"
},
"image-url-list": {
"title": "List of URLs images to download",
"description": "The list shall be list of direct URLs to images, followed by hash (#), then by image MD5SUM. Each image shall appear on newline, like: \"https://example.com/image.iso#06226c7fac5bacfa385872a19bb99684<newline>https://example.com/another-image.iso#31b40d58b18e038498ddb46caea1361c\". They will be provided in KVM image list according to the order on the list. After updating the list, the instance has to be restarted to refresh it. Amount of images is limited to 4, and one image can be maximum 5G. Image will be downloaded and checked against its MD5SUM 4 times, then it will be considered as impossible to download with given MD5SUM. Each image has to be downloaded in time shorter than 4 hours, so in case of very slow images to access, it can take up to 16 hours to download all of them. Note: The instance has to be restarted in order to update the list of available images in the VM.",
"type": "string",
"textarea": "true"
} }
}, },
"type": "object" "type": "object"
...@@ -508,6 +502,60 @@ ...@@ -508,6 +502,60 @@
"description": "If the VM of cluster doesn't run Ansible and report status to this SlapOS instances, then this allow to disable ansible promise so your instance will not fail to check ansible promise.", "description": "If the VM of cluster doesn't run Ansible and report status to this SlapOS instances, then this allow to disable ansible promise so your instance will not fail to check ansible promise.",
"type": "boolean", "type": "boolean",
"default": false "default": false
},
"boot-image-url-list": {
"title": "Boot image list",
"description": "The list shall be list of direct URLs to images, followed by hash (#), then by image MD5SUM. Each image shall appear on newline, like: \"https://example.com/image.iso#06226c7fac5bacfa385872a19bb99684<newline>https://example.com/another-image.iso#31b40d58b18e038498ddb46caea1361c\". They will be provided in KVM image list according to the order on the list. After updating the list, the instance has to be restarted to refresh it. Amount of images is limited to 4, and one image can be maximum 5G. Image will be downloaded and checked against its MD5SUM 4 times, then it will be considered as impossible to download with given MD5SUM. Each image has to be downloaded in time shorter than 4 hours, so in case of very slow images to access, it can take up to 16 hours to download all of them. Note: The instance has to be restarted in order to update the list of available images in the VM.",
"type": "string",
"textarea": "true"
},
"boot-image-url-select": {
"title": "Boot image",
"type": "array",
"oneOf": [
{
"const": [
"https://shacache.nxdcdn.com/0a6aee1d9aafc1ed095105c052f9fdd65ed00ea9274188c9cd0072c8e6838ab40e246d45a1e6956d74ef1b04a1fc042151762f25412e9ff0cbf49418eef7992e#a3ebc76aec372808ad80000108a2593a"
],
"title": "Debian Buster 10.5 netinst x86_64"
},
{
"const": [
"https://shacache.nxdcdn.com/ce5ddfdbdaccdf929b7fe321212356347d82a02f6b7733427282b416f113d91e587682b003e9d376ac189c3b731595c50c236962aadf2720c16d9f36913577c0#23bf2a2d60271e553e63525e794415f1"
],
"title": "Centos 8.2004 Minimal x86_64"
},
{
"const": [
"https://shacache.nxdcdn.com/302c990c6d69575ff24c96566e5c7e26bf36908abb0cd546e22687c46fb07bf8dba595bf77a9d4fd9ab63e75c0437c133f35462fd41ea77f6f616140cd0e5e6a#f3a306f40e4a313fb5a584d73b3dee8f"
],
"title": "Ubuntu Focal 20.04.1 Live Server x86_64"
},
{
"const": [
"https://shacache.nxdcdn.com/6635269a7eb6fbd6b85fda40cd94f14a27bf53cb1fc82ffcce9fe386a025a43e1ab681db7e8cec50416bfbfc90262f0d95273686a101c74b3f17646f0a34c85b#3708a59af6cf820a95cafe0ae73ac399"
],
"title": "openSUSE Leap 15.2 NET x86_64"
},
{
"const": [
"https://shacache.nxdcdn.com/fc17e8c6ae0790162f4beb8fa6226d945cff638429588999b3a08493ff27b280dc2939fba825ae04be1d9082ea8d7c3c002c5e4c39fbbcf88b8ab5104619e28a#ebcdb2223a77f098af3923fe1fa180aa"
],
"title": "Arch Linux 2020.09.01 x86_64"
},
{
"const": [
"https://shacache.nxdcdn.com/c5a511f349a1146b615e6fab9c24f9be4362046adcf24f0ff82c470d361fac5f6628895e2110ebf8ff87db49d4c413a0a332699da6b1bec64275e0c17a15b999#ca7a1e555c04b4d9a549065fa2ddf713"
],
"title": "Fedora Server 32-1.6 netinst x86_64"
},
{
"const": [
"https://shacache.nxdcdn.com/6c355def68b3c0427f21598cb054ffc893568902f205601ac60f192854769b31bc9cff8eeb6ce99ef975a8fb887d8d3e56fc6cd5ea5cb4b3bba1175c520047cb#57088b77f795ca44b00971e44782ee23"
],
"title": "FreeBSD 12.1 RELEASE bootonly x86_64"
}
]
} }
}, },
"type": "object" "type": "object"
......
...@@ -127,9 +127,13 @@ config-document-host = ${apache-conf:ip} ...@@ -127,9 +127,13 @@ config-document-host = ${apache-conf:ip}
config-document-port = ${apache-conf:port} config-document-port = ${apache-conf:port}
config-document-path = ${hash-code:passwd} config-document-path = ${hash-code:passwd}
config-keyboard-layout-language = {{ dumps(kvm_parameter_dict.get('keyboard-layout-language', 'fr')) }} config-keyboard-layout-language = {{ dumps(kvm_parameter_dict.get('keyboard-layout-language', 'fr')) }}
{%- if 'image-url-list' in kvm_parameter_dict %} {%- if 'boot-image-url-list' in kvm_parameter_dict %}
{#- play nice: if parameter was not constructed by the original request, do not send it at all #} {#- play nice: if parameter was not constructed by the original request, do not send it at all #}
config-image-url-list = {{ kvm_parameter_dict['image-url-list'] }} config-boot-image-url-list = {{ kvm_parameter_dict['boot-image-url-list'] }}
{%- endif %}
{%- if 'boot-image-url-select' in kvm_parameter_dict %}
{#- play nice: if parameter was not constructed by the original request, do not send it at all #}
config-boot-image-url-select = {{ kvm_parameter_dict['boot-image-url-select'] }}
{%- endif %} {%- endif %}
config-type = cluster config-type = cluster
......
{ {
"type": "object", "type": "object",
"$schema": "http://json-schema.org/draft-04/schema", "$schema": "http://json-schema.org/draft-06/schema",
"title": "Input Parameters", "title": "Input Parameters",
"properties": { "properties": {
"enable-device-hotplug": { "enable-device-hotplug": {
...@@ -366,11 +366,59 @@ ...@@ -366,11 +366,59 @@
"format": "uri", "format": "uri",
"default": "http://git.erp5.org/gitweb/slapos.git/blob_plain/HEAD:/software/apache-frontend/software.cfg" "default": "http://git.erp5.org/gitweb/slapos.git/blob_plain/HEAD:/software/apache-frontend/software.cfg"
}, },
"image-url-list": { "boot-image-url-list": {
"title": "List of URLs images to download", "title": "Boot image list",
"description": "The list shall be list of direct URLs to images, followed by hash (#), then by image MD5SUM. Each image shall appear on newline, like: \"https://example.com/image.iso#06226c7fac5bacfa385872a19bb99684<newline>https://example.com/another-image.iso#31b40d58b18e038498ddb46caea1361c\". They will be provided in KVM image list according to the order on the list. After updating the list, the instance has to be restarted to refresh it. Amount of images is limited to 4, and one image can be maximum 5G. Image will be downloaded and checked against its MD5SUM 4 times, then it will be considered as impossible to download with given MD5SUM. Each image has to be downloaded in time shorter than 4 hours, so in case of very slow images to access, it can take up to 16 hours to download all of them. Note: The instance has to be restarted in order to update the list of available images in the VM.", "description": "The list shall be list of direct URLs to images, followed by hash (#), then by image MD5SUM. Each image shall appear on newline, like: \"https://example.com/image.iso#06226c7fac5bacfa385872a19bb99684<newline>https://example.com/another-image.iso#31b40d58b18e038498ddb46caea1361c\". They will be provided in KVM image list according to the order on the list. After updating the list, the instance has to be restarted to refresh it. Amount of images is limited to 4, and one image can be maximum 5G. Image will be downloaded and checked against its MD5SUM 4 times, then it will be considered as impossible to download with given MD5SUM. Each image has to be downloaded in time shorter than 4 hours, so in case of very slow images to access, it can take up to 16 hours to download all of them. Note: The instance has to be restarted in order to update the list of available images in the VM.",
"type": "string", "type": "string",
"textarea": "true" "textarea": "true"
},
"boot-image-url-select": {
"title": "Boot image",
"type": "array",
"oneOf": [
{
"const": [
"https://shacache.nxdcdn.com/0a6aee1d9aafc1ed095105c052f9fdd65ed00ea9274188c9cd0072c8e6838ab40e246d45a1e6956d74ef1b04a1fc042151762f25412e9ff0cbf49418eef7992e#a3ebc76aec372808ad80000108a2593a"
],
"title": "Debian Buster 10.5 netinst x86_64"
},
{
"const": [
"https://shacache.nxdcdn.com/ce5ddfdbdaccdf929b7fe321212356347d82a02f6b7733427282b416f113d91e587682b003e9d376ac189c3b731595c50c236962aadf2720c16d9f36913577c0#23bf2a2d60271e553e63525e794415f1"
],
"title": "Centos 8.2004 Minimal x86_64"
},
{
"const": [
"https://shacache.nxdcdn.com/302c990c6d69575ff24c96566e5c7e26bf36908abb0cd546e22687c46fb07bf8dba595bf77a9d4fd9ab63e75c0437c133f35462fd41ea77f6f616140cd0e5e6a#f3a306f40e4a313fb5a584d73b3dee8f"
],
"title": "Ubuntu Focal 20.04.1 Live Server x86_64"
},
{
"const": [
"https://shacache.nxdcdn.com/6635269a7eb6fbd6b85fda40cd94f14a27bf53cb1fc82ffcce9fe386a025a43e1ab681db7e8cec50416bfbfc90262f0d95273686a101c74b3f17646f0a34c85b#3708a59af6cf820a95cafe0ae73ac399"
],
"title": "openSUSE Leap 15.2 NET x86_64"
},
{
"const": [
"https://shacache.nxdcdn.com/fc17e8c6ae0790162f4beb8fa6226d945cff638429588999b3a08493ff27b280dc2939fba825ae04be1d9082ea8d7c3c002c5e4c39fbbcf88b8ab5104619e28a#ebcdb2223a77f098af3923fe1fa180aa"
],
"title": "Arch Linux 2020.09.01 x86_64"
},
{
"const": [
"https://shacache.nxdcdn.com/c5a511f349a1146b615e6fab9c24f9be4362046adcf24f0ff82c470d361fac5f6628895e2110ebf8ff87db49d4c413a0a332699da6b1bec64275e0c17a15b999#ca7a1e555c04b4d9a549065fa2ddf713"
],
"title": "Fedora Server 32-1.6 netinst x86_64"
},
{
"const": [
"https://shacache.nxdcdn.com/6c355def68b3c0427f21598cb054ffc893568902f205601ac60f192854769b31bc9cff8eeb6ce99ef975a8fb887d8d3e56fc6cd5ea5cb4b3bba1175c520047cb#57088b77f795ca44b00971e44782ee23"
],
"title": "FreeBSD 12.1 RELEASE bootonly x86_64"
}
]
} }
} }
} }
This diff is collapsed.
...@@ -24,7 +24,18 @@ if __name__ == "__main__": ...@@ -24,7 +24,18 @@ if __name__ == "__main__":
image_number = 0 image_number = 0
data = fh.read() data = fh.read()
configuration_dict['config-md5sum'] = hashlib.md5(data).hexdigest() configuration_dict['config-md5sum'] = hashlib.md5(data).hexdigest()
for entry in data.decode('utf-8').split(): if source_configuration.endswith('.json'):
data = data.strip()
data_list = []
if len(data):
try:
data_list = json.loads(data)
except Exception:
error_list.append('ERR: Data is not a JSON')
data_list = []
else:
data_list = data.decode('utf-8').split()
for entry in data_list:
split_entry = entry.split('#') split_entry = entry.split('#')
if len(split_entry) != 2: if len(split_entry) != 2:
error_list.append('ERR: entry %r is incorrect' % (entry,)) error_list.append('ERR: entry %r is incorrect' % (entry,))
......
...@@ -99,7 +99,8 @@ enable_device_hotplug = '{{ parameter_dict.get("enable-device-hotplug") }}'.lowe ...@@ -99,7 +99,8 @@ enable_device_hotplug = '{{ parameter_dict.get("enable-device-hotplug") }}'.lowe
logfile = '{{ parameter_dict.get("log-file") }}' logfile = '{{ parameter_dict.get("log-file") }}'
image_url_list_json_config = '{{ parameter_dict.get("image-url-list-json-config") }}' boot_image_url_list_json_config = '{{ parameter_dict.get("boot-image-url-list-json-config") }}'
boot_image_url_select_json_config = '{{ parameter_dict.get("boot-image-url-select-json-config") }}'
if hasattr(ssl, '_create_unverified_context') and url_check_certificate == 'false': if hasattr(ssl, '_create_unverified_context') and url_check_certificate == 'false':
opener = FancyURLopener(context=ssl._create_unverified_context()) opener = FancyURLopener(context=ssl._create_unverified_context())
...@@ -370,24 +371,40 @@ for nbd_ip, nbd_port in nbd_list: ...@@ -370,24 +371,40 @@ for nbd_ip, nbd_port in nbd_list:
'-drive', '-drive',
'file=nbd:[%s]:%s,media=cdrom' % (nbd_ip, nbd_port)]) 'file=nbd:[%s]:%s,media=cdrom' % (nbd_ip, nbd_port)])
else: else:
image_url_list_used = False index = 0
if image_url_list_json_config: if boot_image_url_select_json_config:
# Support image-url-list # Support boot-image-url-select
with open(image_url_list_json_config) as fh: with open(boot_image_url_select_json_config) as fh:
image_config = json.load(fh) image_config = json.load(fh)
if image_config['error-amount'] == 0: if image_config['error-amount'] == 0:
for image in sorted(image_config['image-list'], key=lambda k: k['link']): for image in sorted(image_config['image-list'], key=lambda k: k['link']):
link = os.path.join(image_config['destination-directory'], image['link']) link = os.path.join(image_config['destination-directory'], image['link'])
if os.path.exists(link) and os.path.islink(link): if os.path.exists(link) and os.path.islink(link):
image_url_list_used = True
kvm_argument_list.extend([ kvm_argument_list.extend([
'-drive', '-drive', 'file=%s,media=cdrom,if=none,id=cdrom%s' % (link, index),
'file=%s,media=cdrom' % (link,) '-device', 'virtio-scsi-pci,id=scsi%s' % (index,),
'-device', 'scsi-cd,bus=scsi%s.0,drive=cdrom%s' % (index, index)
]) ])
if not image_url_list_used: index += 1
# If no NBD is specified/available not downloadable image: use internal disk image if boot_image_url_list_json_config:
# Support boot-image-url-list
with open(boot_image_url_list_json_config) as fh:
image_config = json.load(fh)
if image_config['error-amount'] == 0:
for image in sorted(image_config['image-list'], key=lambda k: k['link']):
link = os.path.join(image_config['destination-directory'], image['link'])
if os.path.exists(link) and os.path.islink(link):
kvm_argument_list.extend([
'-drive', 'file=%s,media=cdrom,if=none,id=cdrom%s' % (link, index),
'-device', 'virtio-scsi-pci,id=scsi%s' % (index,),
'-device', 'scsi-cd,bus=scsi%s.0,drive=cdrom%s' % (index, index)
])
index += 1
# Always add by default the default image
kvm_argument_list.extend([ kvm_argument_list.extend([
'-drive', 'file=%s,media=cdrom' % default_cdrom_iso '-drive', 'file=%s,media=cdrom,if=none,id=cdrom%s' % (default_cdrom_iso, index),
'-device', 'virtio-scsi-pci,id=scsi%s' % (index,),
'-device', 'scsi-cd,bus=scsi%s.0,drive=cdrom%s' % (index, index)
]) ])
......
This diff is collapsed.
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment