- 18 Mar, 2025 1 commit
-
-
Xavier Thompson authored
Add option `allow-tcp-connections-on-replica`, set to true by default. This option corresponds to `skip-networking` in mariadb configuration. Changing it only takes effect the next time mariadb is restarted; this does not occur automatically. Instead a script is provided to shutdown mariadb: bin/mariadb-shutdown-gracefully. Mariadb will then be started by SlapOS the next time the instance is processed (even when master is offline). TODO: This is currently sub-optimal because the promise will trigger a bang of the partition when allow-tcp-connections-on-replica is enabled and mariadb has not been restarted, but as SlapOS will not restart the mariadb service automatically, the partition is processed needlessly. Leaving this option set currently limits the ability to takoever fully as a part of the service of a mariadb instance, without SlapOS action: the value of skip-networking will be controlled by SlapOS, rather than by the user interacting with the mariadb service.
-
- 17 Mar, 2025 16 commits
-
-
Xavier Thompson authored
Deactivate zope promises when the neo is expected to be BACKINGUP, as this makes the zope process crash, which is currently expected. Deactivating the zope process entirely in that case is not desired because reactivating it would require updating instance parameters and reprocessing the partition. Instead, ideally, the zope service should adapt to the state of the neo. Also, move zope service from etc/service to etc/run to make it not be "on-watch", so that when the neo is BACKINGUP and zope crashes, the partition does not bang and reprocess continuously. This seems ok because the promise already asserts the service is running.
-
Xavier Thompson authored
-
Xavier Thompson authored
Add scripts in bin directory that can be called manually to apply a new primary-url (after having edited it in the instance parameters) by turning a primary into a replica or by making a replica follow a new primary, all without deleting existing data - so this is useful when the existing data does not diverge from the new primary. Being able to start replicating from a new primary without deleting existing data is convenient, but maybe it's better or sufficient or both to only be able to tell a mariadb to just destroy its existing data and re-bootstrap from scratch. Being able to delete all existing data and start from scratch seems necessary in any case, but it's not clear that a special feature is needed for this: maybe it's completely enough to be able to destroy and re-request the instance through standard slapos operations. Questions remain for automated usage: 1. One option is to provide a url the user may simply interact with (clickable button ...) to call the appropriate script. This kind of interaction could also be used to trigger the deletion of all existing data for a fresh bootstrap. 2. Another option is that editing the primary-url in the parameters results in the appropriate script being called. This makes sense as a way to remain in sync with given parameters, although there are some drawbacks: the uncertain propagation time, and the fact there is no guarantee that the existing data can converge to the state of the new primary. This might be grounds to keep all this strictly manual. In any case, this relies already on the propagation of the instance parameters to input the new primary-url, so it may not be useful at all for any use-case, manual or automated.
-
Xavier Thompson authored
Add a script in bin directory that can be called manually to turn a replica into a primary. Being able to turn a replica into a primary is an absolutely necessary feature of replication. It's a necessary step of a takoever procedure. Questions remain for automatisation. One option is to provide a url the user may simply interact with (clickable button ...) to trigger the script, that may also be used automatically as part of a larger takoever procedure. Triggering the change to primary by editing the instance parameters - such as by removing the replication-related parameters - does not seem best: one immediate drawback is the uncertain propagation time of instance parameters; another is the reliance on the availability of the SlapOS master to trigger a usually time-sensitive operation.
-
Xavier Thompson authored
1. Remove mariadb_update service. Instead, initialize databases and users on creation, and run updater and apply timezones info on every (re)start. This covers the actions that mariadb_update used to handle. 2. Allow requesting a mariadb set-up to replicate another mariadb, using parameters of the form: ``` 'replication': { 'bootstrap-url': 'http(s)://<recent-backup-of-primary> 'primary-url': 'mysql://<replication-user>:<password>@<ip>:<port>' 'seconds-behind-master-threshold': <integer, defaults to 0> } ``` This takes effect on mariadb database creation - when no data exists yet. That way existing data cannot be deleted by setting or changing the replication parameters after the fact. A promise checks that the state of the running mariadb matches the requested state (replica/primary, replication source); but if not, the mariadb database will not automatically converge without human intervention once ~/srv/mariadb directory exists. The bootstrap-url may be omitted: this skips replication bootstrap and requires that all binlogs be still available on the primary. This is useful when the primary is recent and may not have a ready backup for bootstrap yet. To facilitate these changes, component/mariadb now exposes a template script for the mariadbd service, with ready hooks to take actions on database creation and on database (re)start.
-
Xavier Thompson authored
Use a generated password and publish its url.
-
Xavier Thompson authored
-
Xavier Thompson authored
-
Xavier Thompson authored
-
Xavier Thompson authored
When upstream-cluster and upstream-masters are given, also pass --backup to the neo master so that it converges automatically to BACKINGUP state.
-
Xavier Thompson authored
-
Xavier Thompson authored
Remove ``` [gcc:python2] part = gcc-8.5 ``` from neoppod/software-common.cfg and dream/software.cfg as it causes gcc being pinned to a different version before and after rebootstrap when the initial python version is not Python2. This reverts dc91e5d4.
-
Xavier Thompson authored
-
Xavier Thompson authored
When use-ipv6 is enabled, make mariadb publish an url of the form: `mysql://user:password@[ipv6]:port/database` instead of: `mysql://user:password@ipv6:port/database` which results in the zope instances crashing during processing due to urlparse.urlsplit failing to parse the url.
-
Carlos Ramos Carreño authored
The `which` command is not always available, so we use `command -v` instead. An issue was opened upstream in https://github.com/GNUAspell/aspell-lang/issues/3 , and a PR was proposed but not yet accepted. See merge request !1761
-
Nicolas Wavrant authored
-
- 14 Mar, 2025 2 commits
-
-
Thomas Gambier authored
-
Nicolas Wavrant authored
Due to disk usage on my production server, I had to move the theia's backup to a different file system. I bind mount this other fs on: slappart0/srv/backup/theia Mounting on "slappart0/srv/backup" would make logrogate complain (logrotate is used in the "backup" cron, triggering the resiliency process). So "slappart0/srv/backup/theia" was the 2nd best idea. Unfortunately for my case, os.rename raises if the src and dest filesystems are different, as it os.rename guarantees the atomicity of the operation. As atomicity is not that important for copying the signature file, it can be replaced by shutil.move.
-
- 13 Mar, 2025 1 commit
-
-
Kazuhiko Shiozaki authored
-
- 12 Mar, 2025 1 commit
-
-
Thomas Gambier authored
-
- 11 Mar, 2025 3 commits
-
-
Łukasz Nowak authored
In case if there is a problem the output allows to understand the underlying cause. In case of correct run, no output is shown.
-
Łukasz Nowak authored
-
Łukasz Nowak authored
-
- 10 Mar, 2025 4 commits
-
-
Kazuhiko Shiozaki authored
-
Łukasz Nowak authored
It's never used, and that's not the repetetive nor deploy-like way of doing development in the end.
-
Łukasz Nowak authored
This part is unit tested against slapproxy, but unfortunately other implementations of SlapOS Master might return None in such case failing fully the profile. Fix it without test, but with marking the problem.
-
Łukasz Nowak authored
Thanks to this tool allowing incremental checksum calculation increase of backup size does not result with increase of backup time spend on checksum calculation.
-
- 07 Mar, 2025 4 commits
-
-
Boxiang Sun authored
The statsmodels 0.14.4 needs upgraded version of slapos.toolbox. Also fix mmonitor stack eggs version. See merge request !1757
-
Romain Courteaud authored
The value can already be accessed from the buildout parameters. No need to do one http query to read the value.
-
Romain Courteaud authored
-
Jérome Perrin authored
See merge request !1756
-
- 06 Mar, 2025 4 commits
-
-
Boxiang Sun authored
See merge request !1749
-
Boxiang Sun authored
-
Boxiang Sun authored
-
Jérome Perrin authored
The previous approach was ingesting uless "Time" lines sometimes
-
- 05 Mar, 2025 1 commit
-
-
Jérome Perrin authored
the default is 128 lines which is sometimes not enough
-
- 04 Mar, 2025 2 commits
-
-
Kirill Smelkov authored
After previous patch the build of this test passes further than buildout rebootstrap but fails with Installing software release /srv/slapgrid/slappart5/srv/project/slapos/component/ZODB/test-zodb4-wc2.cfg ... ... Installing persistent-repository. Warning: "branch" parameter with value "master" is ignored. Checking out to revision 4.6.4-0-g7ed95cf. Cloning into '/srv/slapgrid/slappart5/srv/runner/software/04a9ddca848c1bb62254aab83f212066/parts/persistent'... HEAD is now at 7ed95cf Preparing release 4.6.4 Updating python-cffi. Updating persistent. DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. pip 21.0 will drop support for Python 2.7 in January 2021. More details about Python 2 support in pip can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support pip 21.0 will remove support for this functionality. Installing BTrees. networkcache: Trying to download pypi:persistent=4.9.3 from network cache... Getting distribution for 'persistent==4.9.3'. ERROR: Command errored out with exit status 1: command: /srv/slapgrid/slappart5/srv/runner/shared/python2.7/474396ed0d332179d494b6d9d3f28ef5/bin/python2.7 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-req-build-kywY6K/setup.py'"'"'; __file__='"'"'/tmp/pip-req-build-kywY6K/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-LeIkZV cwd: /tmp/pip-req-build-kywY6K/ Complete output (25 lines): DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. pip 21.0 will drop support for Python 2.7 in January 2021. More details about Python 2 support in pip can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support pip 21.0 will remove support for this functionality. WARNING: Url 'file:///dev/null/cffi/' is ignored: it is neither a file nor a directory. ERROR: Could not find a version that satisfies the requirement cffi (from versions: none) ERROR: No matching distribution found for cffi Traceback (most recent call last): File "<string>", line 1, in <module> File "/tmp/pip-req-build-kywY6K/setup.py", line 149, in <module> entry_points={}) File "/srv/slapgrid/slappart5/srv/runner/software/04a9ddca848c1bb62254aab83f212066/eggs/setuptools-44.1.1-py2.7.egg/setuptools/__init__.py", line 161, in setup _install_setup_requires(attrs) File "/srv/slapgrid/slappart5/srv/runner/software/04a9ddca848c1bb62254aab83f212066/eggs/setuptools-44.1.1-py2.7.egg/setuptools/__init__.py", line 156, in _install_setup_requires dist.fetch_build_eggs(dist.setup_requires) File "/srv/slapgrid/slappart5/srv/runner/software/04a9ddca848c1bb62254aab83f212066/eggs/setuptools-44.1.1-py2.7.egg/setuptools/dist.py", line 721, in fetch_build_eggs replace_conflicting=True, File "/srv/slapgrid/slappart5/srv/runner/software/04a9ddca848c1bb62254aab83f212066/eggs/setuptools-44.1.1-py2.7.egg/pkg_resources/__init__.py", line 782, in resolve replace_conflicting=replace_conflicting File "/srv/slapgrid/slappart5/srv/runner/software/04a9ddca848c1bb62254aab83f212066/eggs/setuptools-44.1.1-py2.7.egg/pkg_resources/__init__.py", line 1065, in best_match return self.obtain(req, installer) File "/srv/slapgrid/slappart5/srv/runner/software/04a9ddca848c1bb62254aab83f212066/eggs/setuptools-44.1.1-py2.7.egg/pkg_resources/__init__.py", line 1077, in obtain return installer(requirement) File "/srv/slapgrid/slappart5/srv/runner/software/04a9ddca848c1bb62254aab83f212066/eggs/setuptools-44.1.1-py2.7.egg/setuptools/dist.py", line 777, in fetch_build_egg return fetch_build_egg(self, req) File "/srv/slapgrid/slappart5/srv/runner/software/04a9ddca848c1bb62254aab83f212066/eggs/setuptools-44.1.1-py2.7.egg/setuptools/installer.py", line 130, in fetch_build_egg raise DistutilsError(str(e)) distutils.errors.DistutilsError: Command '['/srv/slapgrid/slappart5/srv/runner/shared/python2.7/474396ed0d332179d494b6d9d3f28ef5/bin/python2.7', '-m', 'pip', '--disable-pip-version-check', 'wheel', '--no-deps', '-w', '/tmp/tmptaQnzb', '--quiet', '--index-url', 'file:///dev/null', 'cffi']' returned non-zero exit status 1 ---------------------------------------- ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output. While: Installing BTrees. Base installation request: 'persistent' Getting distribution for 'persistent==4.9.3'. An internal error occurred due to a bug in either zc.buildout or in a recipe being used: Traceback (most recent call last): File "/srv/slapgrid/slappart5/srv/runner/software/04a9ddca848c1bb62254aab83f212066/eggs/zc.buildout-3.0.1+slapos009-py2.7.egg/zc/buildout/buildout.py", line 2664, in main getattr(buildout, command)(args) File "/srv/slapgrid/slappart5/srv/runner/software/04a9ddca848c1bb62254aab83f212066/eggs/zc.buildout-3.0.1+slapos009-py2.7.egg/zc/buildout/buildout.py", line 855, in install self._install_parts(install_args) File "/srv/slapgrid/slappart5/srv/runner/software/04a9ddca848c1bb62254aab83f212066/eggs/zc.buildout-3.0.1+slapos009-py2.7.egg/zc/buildout/buildout.py", line 1026, in _install_parts installed_files = self[part]._call(recipe.install) File "/srv/slapgrid/slappart5/srv/runner/software/04a9ddca848c1bb62254aab83f212066/eggs/zc.buildout-3.0.1+slapos009-py2.7.egg/zc/buildout/buildout.py", line 1948, in _call return f() File "/srv/slapgrid/slappart5/srv/runner/software/04a9ddca848c1bb62254aab83f212066/eggs/zc.recipe.egg-2.0.8.dev0+slapos009-py2.7.egg/zc/recipe/egg/custom.py", line 79, in install self._install_setup_eggs() File "/srv/slapgrid/slappart5/srv/runner/software/04a9ddca848c1bb62254aab83f212066/eggs/zc.recipe.egg-2.0.8.dev0+slapos009-py2.7.egg/zc/recipe/egg/custom.py", line 119, in _install_setup_eggs newest=self.newest, File "/srv/slapgrid/slappart5/srv/runner/software/04a9ddca848c1bb62254aab83f212066/eggs/zc.buildout-3.0.1+slapos009-py2.7.egg/zc/buildout/easy_install.py", line 1178, in install return installer.install(specs, working_set, patch_dict=patch_dict) File "/srv/slapgrid/slappart5/srv/runner/software/04a9ddca848c1bb62254aab83f212066/eggs/zc.buildout-3.0.1+slapos009-py2.7.egg/zc/buildout/easy_install.py", line 835, in install for dist in self._get_dist(requirement, ws): File "/srv/slapgrid/slappart5/srv/runner/software/04a9ddca848c1bb62254aab83f212066/eggs/zc.buildout-3.0.1+slapos009-py2.7.egg/zc/buildout/easy_install.py", line 711, in _get_dist dists = [_move_to_eggs_dir_and_compile(dist, self._dest, self)] File "/srv/slapgrid/slappart5/srv/runner/software/04a9ddca848c1bb62254aab83f212066/eggs/zc.buildout-3.0.1+slapos009-py2.7.egg/zc/buildout/easy_install.py", line 2205, in _move_to_eggs_dir_and_compile [tmp_loc] = call_pip_wheel(dist.location, tmp_dest, options) File "/srv/slapgrid/slappart5/srv/runner/software/04a9ddca848c1bb62254aab83f212066/eggs/zc.buildout-3.0.1+slapos009-py2.7.egg/zc/buildout/easy_install.py", line 2100, in call_pip_wheel call_pip_command(['wheel', '-w', dest], [spec], options) File "/srv/slapgrid/slappart5/srv/runner/software/04a9ddca848c1bb62254aab83f212066/eggs/zc.buildout-3.0.1+slapos009-py2.7.egg/zc/buildout/easy_install.py", line 2038, in call_pip_command subprocess.check_call(args, env=env) File "/srv/slapgrid/slappart5/srv/runner/shared/python2.7/474396ed0d332179d494b6d9d3f28ef5/lib/python2.7/subprocess.py", line 190, in check_call raise CalledProcessError(retcode, cmd) CalledProcessError: Command '['/srv/slapgrid/slappart5/srv/runner/shared/python2.7/474396ed0d332179d494b6d9d3f28ef5/bin/python2.7', '-m', 'pip', 'wheel', '-w', '/srv/slapgrid/slappart5/srv/runner/software/04a9ddca848c1bb62254aab83f212066/eggs/tmpmfO6r5', '--no-deps', '-q', '--no-index', '--no-build-isolation', '--no-python-version-warning', '--no-cache-dir', '/tmp/tmpbuYU1zget_dist/persistent-4.9.3.tar.gz']' returned non-zero exit status 1 2025-03-03 06:25:33 slapos[1517575] ERROR Failed to run buildout profile in directory '/srv/slapgrid/slappart5/srv/runner/software/04a9ddca848c1bb62254aab83f212066' 2025-03-03 06:25:33 slapos[1517575] INFO Finished software releases. What happens here is that test-zodb4-wc2.cfg installs persistent via git clone but at 4.6.4 version, while the version of wanted egg in ZODB/buildout.cfg [versions] is 4.9.3. This way when buildout sees that BTrees needs persistent as the dependency, it looks for "whether we already have persistent==4.9.3 installed", determines the answer as "no" (as we have persistent 4.6.4), and tries to install persistent by itself, and fails somewhere around cffi which we also provide via zc.recipe.egg:custom. The cffi error is only a consequence of original problem of mismatched persistent versions. -> Fix BTrees/persistent build by keeping used persistent versions in sync in between buildout.cfg and test-zodb4-wc2.cfg As test-zodb4-wc2.cfg explains we need persistent to be a git checkout because persistent tests want to discover in-tree files that are not present in persistent egg when it is installed in non-development mode: https://erp5.nexedi.net/test_result_module/20201123-3F859E35/7 (look for "AssertionError: could not find my setup.py") https://github.com/zopefoundation/persistent/blob/4.6.4-0-g7ed95cf/persistent/tests/test_docs.py#L37-L43 and on the other hand we do not use git checkout way uniformly because people wanted to use plain eggs when component/ZODB was bootstrapped. This way we have to use both egg and git checkout and keep their versions in sync. After this patch the build of test-zodb4-wc2 is recovered. /cc @xavier_thompson /reviewed-by @kazuhiko, @levin.zimmermann /helped-and-reviewed-by @jerome /reviewed-on !1752
-
Kirill Smelkov authored
Since c170fe3e (ZODB: Provide ZODB4-wc2 which backports MVCC approach from ZODB5) we have a custom version of ZODB4 with backports from ZODB5 so that wendelin.core can work in environments where we still use ZODB4. In 2022 this test got broken after SlapOS transitioned to use python3 by default and fixed in 5cb11e9e (component/ZODB: keep running the test on python3). Then fixed in 2024 again in e742508b (! component/openssl: add OpenSSL-3.0.13, still using OpenSSL-1.1 for python 2.). However as of today building this test fails with slapos.rebootstrap: ************ REBOOTSTRAP: IMPORTANT NOTICE ************ bin/buildout is being reinstalled right now, as new python: /srv/slapgrid/slappart5/srv/runner/shared/python2.7/b8880e7ab7f4124a466ecb817efb447c/bin/python2.7 is available, and buildout is using another python: /opt/slapos/parts/python3/bin/python3 Buildout will be restarted automatically to have this change applied. ************ REBOOTSTRAP: IMPORTANT NOTICE ************ Traceback (most recent call last): File "<string>", line 1, in <module> File "/srv/slapgrid/slappart5/srv/runner/software/04a9ddca848c1bb62254aab83f212066/eggs/zc.buildout-3.0.1+slapos009-py3.7.egg/zc/__init__.py", line 1, in <module> __import__('pkg_resources').declare_namespace(__name__) File "/srv/slapgrid/slappart5/srv/runner/software/04a9ddca848c1bb62254aab83f212066/eggs/setuptools-67.8.0-py3.7.egg/pkg_resources/__init__.py", line 1423 local = f"sanitized.{_safe_segment(rest)}".strip(".") ^ SyntaxError: invalid syntax While: Installing. An internal error occurred due to a bug in either zc.buildout or in a recipe being used: Traceback (most recent call last): File "/srv/slapgrid/slappart5/srv/runner/software/04a9ddca848c1bb62254aab83f212066/eggs/zc.buildout-3.0.1+slapos009-py3.7.egg/zc/buildout/buildout.py", line 2664, in main getattr(buildout, command)(args) File "/srv/slapgrid/slappart5/srv/runner/software/04a9ddca848c1bb62254aab83f212066/eggs/zc.buildout-3.0.1+slapos009-py3.7.egg/zc/buildout/buildout.py", line 855, in install self._install_parts(install_args) File "/srv/slapgrid/slappart5/srv/runner/software/04a9ddca848c1bb62254aab83f212066/eggs/zc.buildout-3.0.1+slapos009-py3.7.egg/zc/buildout/buildout.py", line 870, in _install_parts self._setup_directories() File "/srv/slapgrid/slappart5/srv/runner/software/04a9ddca848c1bb62254aab83f212066/eggs/slapos.rebootstrap-4.7-py3.7.egg/slapos/rebootstrap/__init__.py", line 58, in wrapper return getattr(self, attr)(*args, **kw) File "/srv/slapgrid/slappart5/srv/runner/software/04a9ddca848c1bb62254aab83f212066/eggs/slapos.rebootstrap-4.7-py3.7.egg/slapos/rebootstrap/__init__.py", line 108, in _setup_directories "zc.buildout.buildout.main()" % (paths, args + ['bootstrap'])]) File "/opt/slapos/parts/python3/lib/python3.7/subprocess.py", line 363, in check_call raise CalledProcessError(retcode, cmd) subprocess.CalledProcessError: Command '['/srv/slapgrid/slappart5/srv/runner/shared/python2.7/b8880e7ab7f4124a466ecb817efb447c/bin/python2.7', '-c', "import sys ; sys.path[0:0]=['/srv/slapgrid/slappart5/srv/runner/software/04a9ddca848c1bb62254aab83f212066/eggs/slapos.libnetworkcache-0.25-py3.7.egg', '/srv/slapgrid/slappart5/srv/runner/software/04a9ddca848c1bb62254aab83f212066/eggs/zc.buildout-3.0.1+slapos009-py3.7.egg', '/srv/slapgrid/slappart5/srv/runner/software/04a9ddca848c1bb62254aab83f212066/eggs/setuptools-67.8.0-py3.7.egg', '/srv/slapgrid/slappart5/srv/runner/software/04a9ddca848c1bb62254aab83f212066/eggs/wheel-0.41.2-py3.7.egg', '/srv/slapgrid/slappart5/srv/runner/software/04a9ddca848c1bb62254aab83f212066/eggs/pip-23.2.1-py3.7.egg'] ; import zc.buildout.buildout ; sys.argv[1:1]=['buildout:extends-cache=/tmp/tmpdx2upamb', 'buildout:directory=/srv/slapgrid/slappart5/srv/runner/software/04a9ddca848c1bb62254aab83f212066', 'bootstrap'] ; zc.buildout.buildout.main()"]' returned non-zero exit status 1. The problem here is that even though we use py2 for the software, the used version of setuptools is py3 only and so it breaks. This problem was already cared in f5b80301 (Downgrade pip, wheel and setuptools in python2 SRs), but that commit missed to update test-zodb4-wc2.cfg . -> Fix that. After this patch the build of component/ZODB/test-zodb4-wc2.cfg becomes recovered in the sense that it passes further than rebootstrap, but breaks at the build of persistent / BTrees eggs. We will fix this in the next patch. /cc @xavier_thompson /reviewed-by @jerome, @kazuhiko, @levin.zimmermann /reviewed-on !1752
-
- 03 Mar, 2025 1 commit
-
-
Kazuhiko Shiozaki authored
-