Skip to content

N
nxd-bom
Commit 9273cfb9 authored by Kirill Smelkov's avatar Kirill Smelkov
Browse files
Options

nxd-bom: First draft 

Program nxd-bom is intended to help generating bill of material for a
software-release or slapos node.

Generation of the BOM for a software-release is already draftly
implemented. Generation of the BOM for slapos node is thought-out, but
not currently implemented.

In provided example/ors-bom.txt it could be seen how both parts should
look like. And nxd-bom documents in corresponding TODO section how
BOM(slapos-node) should be computed.

I originally used this draft script to build ors-bom.txt as requested by
a customer, and I no longer actively work on it. However I suggest we
incrementally improve it as needed and maybe also integrate it to be run
automatically in _all_ our builds, so that whenever an SR is built, the
BOM.txt is also built automatically for it.
parent 677f19be
Hide whitespace changes
Inline Side-by-side
Showing with 648 additions and 0 deletions
example/ors-bom.txt 0 → 100644
View file @ 9273cfb9
Bill of materials for ORS software
==================================
ORS software consists of
1) Base OS;
2) Amarisoft stack;
3) Slapos-node package;
4) ors-amarisoft software-release.
Below we describe requirements and options for the Base OS, and provide bill of
materials for components provided by Xunkongjian:
Base OS + Amarisoft stack
-------------------------
For Base OS Amarisoft stack requirements are:
A 64 bit Linux distribution. Fedora 34 is the officially supported distribution.
The following distributions are known as compatible:
Fedora 22 to 34
Cent OS 7
Ubuntu 14 to 20
In Nexedi we usually deploy Debian as the base OS, but for ORS, taking
Amarisoft recommendations into account, we currently deploy Ubuntu 20.04 .
Amarisoft stack details are listed in its documentation.
BOM(ors-amarisoft)
------------------
Python 2.7.18 http://www.python.org/ftp/python/2.7.18/Python-2.7.18.tar.xz
apr 1.7.0 https://archive.apache.org/dist/apr/apr-1.7.0.tar.bz2
apr-util 1.6.1 https://archive.apache.org/dist/apr/apr-util-1.6.1.tar.bz2
autoconf 2.71 http://ftp.gnu.org/gnu/autoconf/autoconf-2.71.tar.gz
automake 1.16.5 https://ftp.gnu.org/gnu/automake/automake-1.16.5.tar.xz
bash 5.1 https://ftp.gnu.org/pub/gnu/bash/bash-5.1.tar.gz
bzip2 1.0.8 https://fossies.org/linux/misc/bzip2-1.0.8.tar.gz
ca-certificates 20210119 http://deb.debian.org/debian/pool/main/c/ca-certificates/ca-certificates_20210119.tar.xz
config 5e531d391852a54e7fab2d8ff55625fca514b305 https://git.savannah.gnu.org/gitweb/?p=config.git;a=snapshot;h=5e531d391852a54e7fab2d8ff55625fca514b305;sf=tgz
coreutils 9.0 https://ftp.gnu.org/gnu/coreutils/coreutils-9.0.tar.xz
curl 7.79.1 http://curl.haxx.se/download/curl-7.79.1.tar.xz
dash 0.5.11 http://gondor.apana.org.au/~herbert/dash/files/dash-0.5.11.tar.gz
dcron 4.5 http://www.jimpryor.net/linux/releases/dcron-4.5.tar.gz
expat 2.2.10 https://github.com/libexpat/libexpat/releases/download/R_2_2_10/expat-2.2.10.tar.lz
file 5.41 http://ftp.icm.edu.pl/packages/file/file-5.41.tar.gz
gdbm 1.22 http://ftp.gnu.org/gnu/gdbm/gdbm-1.22.tar.gz
gettext 0.21 http://ftp.gnu.org/pub/gnu/gettext/gettext-0.21.tar.lz
gmp 6.2.1 https://gmplib.org/download/gmp/gmp-6.2.1.tar.xz
httpd 2.4.54 https://archive.apache.org/dist/httpd/httpd-2.4.54.tar.bz2
libffi 3.3 http://sourceware.org/pub/libffi/libffi-3.3.tar.gz
libtool 2.4.6 http://ftp.gnu.org/gnu/libtool/libtool-2.4.6.tar.xz
libxml2 2.9.14 https://download.gnome.org/sources/libxml2/2.9/libxml2-2.9.14.tar.xz
libxslt 1.1.35 https://download.gnome.org/sources/libxslt/1.1/libxslt-1.1.35.tar.xz
logrotate 3.18.1 https://github.com/logrotate/logrotate/releases/download/3.18.1/logrotate-3.18.1.tar.xz
lunzip 1.13 http://download-mirror.savannah.gnu.org/releases/lzip/lunzip/lunzip-1.13.tar.gz
lz4 1.9.3 https://github.com/lz4/lz4/archive/v1.9.3.tar.gz
m4 1.4.19 http://ftp.gnu.org/gnu/m4/m4-1.4.19.tar.xz
ncurses 6.2 http://ftp.gnu.org/gnu/ncurses/ncurses-6.2.tar.gz
nghttp2 1.40.0 https://github.com/nghttp2/nghttp2/archive/v1.40.0.tar.gz
openssl 1.1.1o https://www.openssl.org/source/openssl-1.1.1o.tar.gz
patch 2.7.6 http://ftp.gnu.org/gnu/patch/patch-2.7.6.tar.xz
pcre 8.45 https://download.sourceforge.net/pcre/pcre/8.45/pcre-8.45.tar.bz2
perl 5.28.1 http://www.cpan.org/src/5.0/perl-5.28.1.tar.xz
pkg-config 0.29.2 https://pkgconfig.freedesktop.org/releases/pkg-config-0.29.2.tar.gz
popt 1.18 http://ftp.rpm.org/popt/releases/popt-1.x/popt-1.18.tar.gz
readline 8.1 http://ftp.gnu.org/gnu/readline/readline-8.1.tar.gz
sqlite-autoconf 3370200 https://sqlite.org/2022/sqlite-autoconf-3370200.tar.gz
util-linux 2.37.2 https://www.kernel.org/pub/linux/utils/util-linux/v2.37/util-linux-2.37.2.tar.xz
xz 5.2.5 https://tukaani.org/xz/xz-5.2.5.tar.bz2
yaml 0.2.5 http://pyyaml.org/download/libyaml/yaml-0.2.5.tar.gz
zlib 1.2.12 https://www.zlib.net/zlib-1.2.12.tar.gz
zstd 1.4.9 https://github.com/facebook/zstd/releases/download/v1.4.9/zstd-1.4.9.tar.gz
>>> eggs:
Jinja2 2.11.3 https://pypi.org/project/Jinja2/2.11.3/
PyRSS2Gen 1.1 https://pypi.org/project/PyRSS2Gen/1.1/
PyYAML 5.4.1 https://pypi.org/project/PyYAML/5.4.1/
backports.lzma 0.0.14 https://pypi.org/project/backports.lzma/0.0.14/
cffi 1.14.0 https://pypi.org/project/cffi/1.14.0/
cns.recipe.symlink 0.2.3 https://pypi.org/project/cns.recipe.symlink/0.2.3/
collective.recipe.template 2.0 https://pypi.org/project/collective.recipe.template/2.0/
cryptography 3.3.2 https://pypi.org/project/cryptography/3.3.2/
lxml 4.6.5 https://pypi.org/project/lxml/4.6.5/
plone.recipe.command 1.1 https://pypi.org/project/plone.recipe.command/1.1/
psutil 5.8.0 https://pypi.org/project/psutil/5.8.0/
pycurl 7.43.0 https://pypi.org/project/pycurl/7.43.0/
pyrsistent 0.16.1 https://pypi.org/project/pyrsistent/0.16.1/
slapos.cookbook 1.0.253 https://pypi.org/project/slapos.cookbook/1.0.253/
slapos.toolbox 0.128 https://pypi.org/project/slapos.toolbox/0.128/
BOM(slapos-node)
----------------
The bill of materials for slapos-node package consists of two parts:
1. list of software that slapos-node brings with itself;
2. list of Base OS packages that slapos-node requests to be installed when deployed on a minimal base system.
List of software that slapos-node brings with itself
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
autoconf 2.71
automake 1.16.5
bison 3.8.2
bzip2 1.0.8
ca-certificates 20210119
firewalld 0.8.6
gettext 0.21
glib 2.56.4
gnu-config 5e531d391852a54e7fab2d8ff55625fca514b305
gobject-introspection 1.50.0
intltool 0.51.0
iptables 1.8.7
libtool 2.4.6
libxml2 2.9.14
ncurses 6.2
nftables 0.9.8
openssl 1.1.1o
perl-Data-Dump 1.22
perl-Encode-Locale 1.03
perl-File-Listing 6.04
perl-HTML-Parser 3.71
perl-HTML-Tagset 3.20
perl-HTTP-Cookies 6.01
perl-HTTP-Daemon 6.01
perl-HTTP-Date 6.02
perl-HTTP-Message 6.06
perl-HTTP-Negotiate 6.01
perl-IO-HTML 1.00
perl-LWP-MediaTypes 6.02
perl-Net-HTTP 6.06
perl-URI 1.76
perl-WWW-RobotRules 6.02
perl-XML-Parser 2.46
perl-libwww-perl 6.06
perl 5.28.1
Python 3.7.7
swig 3.0.12
xz 5.2.5
>>> eggs:
CacheControl 0.12.6
Flask 1.1.2
Jinja2 2.11.3
MarkupSafe 1.0
PyYAML 5.4.1
Werkzeug 2.0.2
attrs 18.2.0
certifi 2020.6.20
cffi 1.14.0
charset_normalizer 2.0.12
click 8.1.3
cliff 2.8.3
cmd2 0.7.0
collective.recipe.template 2.0
cryptography 3.3.2
dbus-python 1.2.16
decorator 4.3.0
distro 1.7.0
idna 2.9
importlib_metadata 1.7.0
itsdangerous 0.24
jsonschema 3.0.2
lockfile 0.12.2
lxml 4.6.5
msgpack 0.6.2
netaddr 0.7.19
netifaces 0.10.7
pbr 2.0.0
plone.recipe.command 1.1
prettytable 0.7.2
psutil 5.8.0
pyOpenSSL 19.1.0
pycparser 2.20
pygobject 3.22.0
pyparsing 2.2.0
pyrsistent 0.16.1
python-slip 0.6.5
requests 2.27.1
six 1.12.0
slapos.core 1.7.10
slapos.libnetworkcache 0.25
slapos.rebootstrap 4.5
slapos.recipe.build 0.55
slapos.recipe.cmmi 0.19
stevedore 1.21.0
supervisor 4.1.0
uritemplate 3.0.0
urllib3 1.26.9
xml_marshaller 1.0.2
zc.buildout 2.7.1+slapos019
zc.recipe.egg 2.0.3+slapos003
zipp 1.2.0
zope.interface 4.3.3
List of base-OS packages installed by https://deploy.rapid.space/slapos
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Debian GNU/Linux 11 additional Base-OS packages installed by slapos-node are:
ansible 2.10.7+merged+base+2.10.8+dfsg-1
binutils 2.35.2-2
binutils-common:amd64 2.35.2-2
binutils-x86-64-linux-gnu 2.35.2-2
build-essential 12.9
bzip2 1.0.8-4
cpp 4:10.2.1-1
cpp-10 10.2.1-6
dirmngr 2.2.27-2+deb11u2
distro-info-data 0.51+deb11u2
dpkg-dev 1.20.11
fakeroot 1.25.3-1.1
fontconfig-config 2.13.1-4.2
fonts-dejavu-core 2.37-2
g++ 4:10.2.1-1
g++-10 10.2.1-6
gcc 4:10.2.1-1
gcc-10 10.2.1-6
gnupg 2.2.27-2+deb11u2
gnupg-l10n 2.2.27-2+deb11u2
gnupg-utils 2.2.27-2+deb11u2
gpg 2.2.27-2+deb11u2
gpg-agent 2.2.27-2+deb11u2
gpg-wks-client 2.2.27-2+deb11u2
gpg-wks-server 2.2.27-2+deb11u2
gpgconf 2.2.27-2+deb11u2
gpgsm 2.2.27-2+deb11u2
ieee-data 20210605.1
javascript-common 11+nmu1
libalgorithm-diff-perl 1.201-1
libalgorithm-diff-xs-perl 0.04-6+b1
libalgorithm-merge-perl 0.08-3
libasan6:amd64 10.2.1-6
libassuan0:amd64 2.5.3-7.1
libatomic1:amd64 10.2.1-6
libbinutils:amd64 2.35.2-2
libbrotli1:amd64 1.0.9-2+b2
libc-dev-bin 2.31-13+deb11u3
libc-devtools 2.31-13+deb11u3
libc6-dev:amd64 2.31-13+deb11u3
libcbor0:amd64 0.5.0+dfsg-2
libcc1-0:amd64 10.2.1-6
libcrypt-dev:amd64 1:4.4.18-4
libctf-nobfd0:amd64 2.35.2-2
libctf0:amd64 2.35.2-2
libdeflate0:amd64 1.7-1
libdpkg-perl 1.20.11
libevent-core-2.1-7:amd64 2.1.12-stable-1
libevent-pthreads-2.1-7:amd64 2.1.12-stable-1
libexpat1:amd64 2.2.10-2+deb11u3
libexpat1-dev:amd64 2.2.10-2+deb11u3
libfakeroot:amd64 1.25.3-1.1
libfido2-1:amd64 1.6.0-2
libfile-fcntllock-perl 0.22-3+b7
libfontconfig1:amd64 2.13.1-4.2
libfreetype6:amd64 2.10.4+dfsg-1+deb11u1
libfuse2:amd64 2.9.9-5
libgcc-10-dev:amd64 10.2.1-6
libgd3:amd64 2.3.0-2
libgdbm-compat4:amd64 1.19-2
libgdbm6:amd64 1.19-2
libgomp1:amd64 10.2.1-6
libisl23:amd64 0.23-1
libitm1:amd64 10.2.1-6
libjbig0:amd64 2.1-3.1+b2
libjpeg62-turbo:amd64 1:2.0.6-4
libjs-jquery 3.5.1+dfsg+~3.5.5-7
libjs-sphinxdoc 3.4.3-2
libjs-underscore 1.9.1~dfsg-3
libksba8:amd64 1.5.0-3
libldap-2.4-2:amd64 2.4.57+dfsg-3+deb11u1
libldap-common 2.4.57+dfsg-3+deb11u1
liblsan0:amd64 10.2.1-6
libmpc3:amd64 1.2.0-1
libmpdec3:amd64 2.5.1-1
libmpfr6:amd64 4.1.0-3
libnpth0:amd64 1.6-3
libnsl-dev:amd64 1.3.0-2
libopts25:amd64 1:5.18.16-4
libperl5.32:amd64 5.32.1-4+deb11u2
libpng16-16:amd64 1.6.37-3
libpython3-dev:amd64 3.9.2-3
libpython3-stdlib:amd64 3.9.2-3
libpython3.9:amd64 3.9.2-1
libpython3.9-dev:amd64 3.9.2-1
libpython3.9-minimal:amd64 3.9.2-1
libpython3.9-stdlib:amd64 3.9.2-1
libquadmath0:amd64 10.2.1-6
libsasl2-2:amd64 2.1.27+dfsg-2.1+deb11u1
libsasl2-modules:amd64 2.1.27+dfsg-2.1+deb11u1
libsasl2-modules-db:amd64 2.1.27+dfsg-2.1+deb11u1
libsensors-config 1:3.6.0-7
libsensors5:amd64 1:3.6.0-7
libsqlite3-0:amd64 3.34.1-3
libstdc++-10-dev:amd64 10.2.1-6
libtiff5:amd64 4.2.0-1+deb11u1
libtirpc-dev:amd64 1.3.1-1
libtsan0:amd64 10.2.1-6
libubsan1:amd64 10.2.1-6
libwebp6:amd64 0.6.1-2.1
libx11-6:amd64 2:1.7.2-1
libx11-data 2:1.7.2-1
libxau6:amd64 1:1.0.9-1
libxcb1:amd64 1.14-3
libxdmcp6:amd64 1:1.1.2-3
libxext6:amd64 2:1.3.3-1.1
libxmuu1:amd64 2:1.1.2-2+b3
libxpm4:amd64 1:3.5.12-1
libyaml-0-2:amd64 0.2.2-1
linux-libc-dev:amd64 5.10.127-1
lm-sensors 1:3.6.0-7
lsb-release 11.1.0
make 4.3-4.1
manpages 5.10-1
manpages-dev 5.10-1
media-types 4.0.0
ntp 1:4.2.8p15+dfsg-1
openssh-client 1:8.4p1-5+deb11u1
patch 2.7.6-7
perl 5.32.1-4+deb11u2
perl-modules-5.32 5.32.1-4+deb11u2
pinentry-curses 1.1.0-4
python-apt-common 2.2.1
python-pip-whl 20.3.4-4+deb11u1
python3 3.9.2-3
python3-apt 2.2.1
python3-argcomplete 1.8.1-1.5
python3-certifi 2020.6.20-1
python3-cffi-backend:amd64 1.14.5-1
python3-chardet 4.0.0-1
python3-cryptography 3.3.2-1
python3-dev 3.9.2-3
python3-distutils 3.9.2-1
python3-dnspython 2.0.0-1
python3-httplib2 0.18.1-3
python3-idna 2.10-1
python3-jinja2 2.11.3-1
python3-jmespath 0.10.0-1
python3-kerberos 1.1.14-3.1+b3
python3-lib2to3 3.9.2-1
python3-libcloud 3.2.0-2
python3-lockfile 1:0.12.2-2.2
python3-markupsafe 1.1.1-1+b3
python3-minimal 3.9.2-3
python3-netaddr 0.7.19-5
python3-ntlm-auth 1.4.0-1
python3-packaging 20.9-2
python3-pip 20.3.4-4+deb11u1
python3-pkg-resources 52.0.0-4
python3-pycryptodome 3.9.7+dfsg1-1+b2
python3-pyparsing 2.4.7-1
python3-requests 2.25.1+dfsg-2
python3-requests-kerberos 0.12.0-2
python3-requests-ntlm 1.1.0-1.1
python3-requests-toolbelt 0.9.1-1
python3-selinux 3.1-3
python3-setuptools 52.0.0-4
python3-simplejson 3.17.2-1
python3-six 1.16.0-2
python3-urllib3 1.26.5-1~exp1
python3-wheel 0.34.2-1
python3-winrm 0.3.0-2
python3-xmltodict 0.12.0-2
python3-yaml 5.3.1-5
python3.9 3.9.2-1
python3.9-dev 3.9.2-1
python3.9-minimal 3.9.2-1
slapos-node 1.7.10+1.0.253+1-1
sntp 1:4.2.8p15+dfsg-1
ucf 3.0043
uml-utilities 20070815.4-1
unzip 6.0-26
xauth 1:1.1-1
xz-utils 5.2.5-2.1~deb11u1
zlib1g-dev:amd64 1:1.2.11.dfsg-2+deb11u1
nxd-bom 0 → 100755
View file @ 9273cfb9
#!/usr/bin/env python
# -*- coding: utf-8 -*-
# Copyright (C) 2022 Nexedi SA and Contributors.
# Kirill Smelkov <kirr@nexedi.com>
#
# This program is free software: you can Use, Study, Modify and Redistribute
# it under the terms of the GNU General Public License version 3, or (at your
# option) any later version, as published by the Free Software Foundation.
#
# You can also Link and Combine this program with other software covered by
# the terms of any of the Free Software licenses or any of the Open Source
# Initiative approved licenses and Convey the resulting work. Corresponding
# source of such a combination shall include the source code for all other
# software used.
#
# This program is distributed WITHOUT ANY WARRANTY; without even the implied
# warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
#
# See COPYING file for full licensing terms.
# See https://www.nexedi.com/licensing for rationale and options.
"""Program nxd-bom generates bill of material for a software-release or slapos node.
Usage: nxd-bom software <path-to-installed-software>
nxd-bom node <slapos deploy-script>
An example of generated bill of material is provided in example/ors-bom.txt .
"""
from __future__ import print_function
import sys, configparser, re
from os.path import basename
from glob import glob
from collections import namedtuple
# PkgInfo represents information about a package
PkgInfo = namedtuple('PkgInfo', [
'name',
'version',
'kind',
'url'])
# bom_software retrieves BOM from .installed.cfg generated by buildout along the build.
def bom_software(installed_software_path): # -> {} name -> PkgInfo
bom = {}
def addbom(urlpath, kind):
name, ver = namever(urlpath)
if '//' in urlpath:
url = urlpath
else:
if kind == 'egg':
# XXX not strictly correct -> better retrieve the actual URL, but buildout does not save it in installed.cfg
url = 'https://pypi.org/project/%s/%s/' % (name, ver)
else:
raise NotImplementedError('TODO url for kind %s' % kind)
info = PkgInfo(name, ver, kind, url)
if name in bom:
assert bom[name] == info, (bom[name], info)
else:
bom[name] = info
idb = configparser.ConfigParser()
idb.read('%s/.installed.cfg' % installed_software_path)
for s in idb.sections():
if s == 'buildout':
continue # [buildout] is used internally
part = idb[s]
recipe = part['recipe']
if recipe == 'slapos.recipe.cmmi':
addbom(part['url'], '') # XXX detect kind?
elif recipe == 'slapos.recipe.build':
# slapos.recipe.build is often used in creative ways to actually
# run python code during the build. Let's detect this via lack of
# `url`, but be careful and explicitly list name of such sections
# not to miss a normal part.
url = part.get('url', None)
if url is None:
if s == 'gcc': # [gcc] is generic section to either use system gcc, or install one via slapos
continue # the one installed via slapos will come in its own section
if s == 'python': # [python] is similar - it picks up either python2 or python3
continue
raise NotImplementedError('%s uses %s without url' % (s, recipe))
else:
addbom(url, '') # XXX detect kind?
elif recipe == 'slapos.recipe.build:download':
# slapos.recipe.build:download is often used to download .conf files, but sometimes it is used to download e.g. binaries
# skip the part, if we can detect that downloaded item is a configuration file
url = part['url']
if isconf(url):
continue
# TODO binary -> parse name/ver
raise NotImplementedError('%s uses %s with url that does not look like a .conf file: %s' % (s, recipe, url))
elif recipe.startswith('slapos.recipe.template'):
url = part.get('url', None)
if url is not None:
if isconf(part['url']):
continue
# NOTE binary is not expected with slapos.recipe.template
raise ValueError('%s uses %s with url that does not look like a .conf file: %s' % (s, recipe, url))
else:
# it is an inline= script
assert 'inline' in part, part
elif recipe == 'zc.recipe.egg:custom':
eggpath = part['__buildout_installed__']
assert len(eggpath.split()) == 1, eggpath # no spaces inside - just one item
addbom(eggpath, 'egg')
elif recipe == 'zc.recipe.egg':
# XXX sadly zc.recipe.egg neither saves in .installed.cfg information about where the eggs are installed,
# nor there are [versions] save.
# -> look the egg on the filesystem
# TODO it is better to fix zc.recipe.egg to save the full information, so that we can build BOM just from .installed.cfg
eggdir = part['_e']
eggdev = part['_d']
eggs = part['eggs'].split()
for eggname in eggs:
eggv = glob('%s/%s-*.egg' % (eggdir, eggname))
eggv += glob('%s/%s-*.egg' % (eggdev, eggname))
if len(eggv) == 0:
raise ValueError('egg %s not found' % egg)
if len(eggv) > 1:
raise ValueError('egg %s is present multiple times: %s' % (egg, eggv))
addbom(eggv[0], 'egg')
else:
raise NotImplementedError('TODO: add support for recipe %s' % recipe)
return bom
def bom_node(XXX):
1/0
# TODO bom_node should:
#
# 1. retrieve list of software that slapos-node brings with itself
# since slapos-node is built via buildout this could be extracted via
# node_software from /opt/slapos/.installed.cfg once we fix slapos-node
# package to preserve that .installed.cfg in the package itself.
#
# 2. retrieve list of base-OS packages that are installed by https://deploy.rapid.space/slapos
# This can be computed as the difference in between two sets of base-OS packages:
# b) set of base-OS packages after deploy.rapid.space/slapos was run, and
# a) set of base-OS packages after minimal base-OS was installed.
#
# this computation could be automated via
# - preparing e.g. Debian 11 minimal chroot via debootstrap,
# - taking snapshot for list of installed packages via `dpkg -l`,
# - then deploying slapos-node in the chroot,
# - taking snapshot for the list of installed package again,
# - and finally computing the difference.
#
# Please see example/ors-bom.txt for "BOM(slapos-node)", "List of software
# that slapos-node brings with itself" and "List of base-OS packages
# installed by https://deploy.rapid.space/slapos" sections for what end result could look like.
# namever extracts item name and version from an url/path.
# examples:
#
# http://www.python.org/ftp/python/2.7.18/Python-2.7.18.tar.xz -> ('Python', '2.7.18')
# https://git.savannah.gnu.org/gitweb/?p=config.git;a=snapshot;h=5e531d39;sf=tgz -> ('config', '5e531d39')
# https://github.com/nghttp2/nghttp2/archive/v1.40.0.tar.gz -> ('nghttp2', '1.40.0')
_gitweb_re = re.compile(r'/gitweb/\?p=(?P<name>\w+)\.git;a=snapshot;h=(?P<rev>\w+)')
_github_re = re.compile(r'github.com/\w+/(?P<name>\w+)/archive/(?P<rev>.+)$')
def namever(url): # -> (name, ver)
for tail in ('.tgz', '.tar.gz', '.tbz', '.tar.bz2', '.tar.xz', '.tar.lz',
'-py2.7.egg', '-py2.7-linux-x86_64.egg'): # FIXME -> re to cover py2/py3 x86/arm ...
url = removesuffix(url, tail)
name, ver = _namever(url)
ver = removeprefix(ver, 'v')
return name, ver
def _namever(url):
for r in (_gitweb_re, _github_re):
m = r.search(url)
if m is not None:
return m.group('name'), m.group('rev')
filename = basename(url)
# re.rsplit([-_], filename, 1)
m = re.search('[-_][^-_]*$', filename)
assert m is not None
name = filename[:m.start()]
ver = filename[m.start()+1:]
return name, ver
# isconf returns whether url points to data related to configuration file
# (contrary to binary executable), or a simple/small script.
def isconf(url):
if url.endswith('.conf.in') or \
url.endswith('.cfg') or \
url.endswith('.cfg.in') or \
url.endswith('.cfg.jinja2.in') or \
url.endswith('.jinja2.cfg') or \
url.endswith('.asn') or \
url.endswith('/ltelogs.jinja2.sh') or \
'/promise/' in url or \
url.endswith('/templates/wrapper.in') or \
url.endswith('/logrotate_entry.in') :
return True
return False
# removeprefix/removesuffix provides fallbacks similarly-named str methods.
def removeprefix(s, prefix):
if s.startswith(prefix):
s = s[len(prefix):]
return s
def removesuffix(s, suffix):
if s.endswith(suffix):
s = s[:len(s)-len(suffix)]
return s
# ----------------------------------------
def main():
if len(sys.argv) != 3 or sys.argv[1] not in ('software', 'node'):
print(__doc__, file=sys.stderr)
sys.exit(2)
what, arg = sys.argv[1:]
if what == 'software':
bom = bom_software(arg)
elif what == 'node':
bom = bom_node(arg)
# print retrieved BOM grouped by kind
kinds = set()
for info in bom.values():
kinds.add(info.kind)
for kind in sorted(kinds):
if kind != '':
print('\n>>> %ss:' % kind)
for name in sorted(bom):
info = bom[name]
if info.kind == kind:
# TODO autoalign
print('%-28s %-10s %s' % (name, info.version, info.url))
if __name__ == '__main__':
main()
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment
GitLab Nexedi Edition | About GitLab | About Nexedi | 沪ICP备2021021310号-2 | 沪ICP备2021021310号-7