A tool to work on backup signatures of created backups in fast way.

See merge request !141

Refactor ARIMA calls and correctly specify the minimum version in
setup.py

Replace deprecated ssl.wrap_socket with ssl.SSLContext
This ensures the code can work with Python3.12

Sometimes, it seems the file analyzed can contain non-ASCII characters, giving
errors such as:

```
codec can't decode byte 0xc3 in position 5138: ordinal not in range(128)
```

It seems there is no reason to not support UTF8, and more generally ignore
encoding errors here. This change has been applied for a few weeks on Nexedi
ERP5 production without problems.

/reviewed-by @tomo @jerome
/reviewed-on !140
/cc @xavier_thompson

See !138

See !138

Also add a comment when we really don't want an AnomalyResult.

See merge request !137

Floating point arithmetics often lead to unsignificant differences between
expected and computed values (this is the typical example of `0.1 + 0.2 !=
0.3`). Because of this, some tests using `assertEqual` to compare two floats
were failing. The Python method
[`assertAlmostEqual`](https://docs.python.org/2.7/library/unittest.html#unittest
.TestCase.assertAlmostEqual) can prevent these issues by comparing values up to
the seventh decimal place. To find all occurences, two RegExp were ran on all
files of the `tests/` folder: `assertEquals?(\\(.+, [0-9]+\\.[0-9]+\\))` and
`assertEquals?(\\([0-9]+\\.[0-9]+)`.

/cc @xavier_thompson @tomo
/reviewed-by @jerome
/reviewed-on !136 slapos.core!719

In that release rf_info started to emit output with "Clock tune:" entry
not indented and so our code was ignoring every line after that leading
to missing to load CPR_option entry and erroring with "no CPRI entry" as
the result.

-> Fix that by adding a quirk to treat "Clock tune:" line specially so
   that processing of per sdr/port records does not stop on it.

/cc @lu.xu
/proposed-for-review-on !135

Fail early in case of bad promise configuration.

Allow to configure periodicity and anomaly in the profile, as otherwise the
promise configuration is not flexible enough for the profile developer, which
has to decide what shall be the effect of promise failure and when it shall
be checked.

Switch to - separated variables, as expected in the profiles.

Honor result_count and failure_amount in the test method too.

Allow to configure result_count and failure_amount.

Old Amarisoft versions emit data that is different compared to what 640c0130 expected:

- there can be trailing spaces after /dev/sdrX@Y
  fix: adjust /dev/sdr regex
- there is empty line coming in the end that fas failing in l.split(':', 1)
  fix: ignore empty lines and make detection of : in the line more
       robust with reporting which line is not valid instead of just
       "ERROR not enough values to unpack (expected 2, got 1)"
- there is CPRI key instead of CPRI_option, and the data in both are different.
  fix: detect automatically whether it is CPRI_option or CPRI option that is present.

/co-authored-by lu.xu <lu.xu@nexedi.com>
/reviewed-on !132

Also improve tests to minimize assertions and make them more readable,
including separating specific tests from global ones, which will make
it much easier to improve the coverage.

When testing is True we do not have hardware around and only check
generated configuration files. Most lopcomm promises were already
skipping themselves under testing=True environment, but
check_lopcomm_config_log and check_lopcomm_stats_log were missing
testing checks. This leads to test failures when trying to test
ors-amarisoft with Lopcomm driver in my upcoming work via [1] e.g. as

    ================================================================================
    Error with promises for the following partitions:
      TestENB_Lopcomm4-0[enb]: Promise 'RU1-stats-log.py' failed with output: Not subscribed

-> Fix it by making sure `testing==True` checks are present in all
   promises related to lopcomm.

P.S. in current ors-amarisoft Lopcomm driver is not tested at all: even
though we have testFDD-LOPCOMM.py there[2] `python unittest discover ...`
does not pick up test files with dash in their name. This is also
true for all other dashed test files. Currently only testTDD.py and
testFDD.py are run during the tests there.

[1] https://lab.nexedi.com/kirr/slapos/blob/5b8ec2b3/software/ors-amarisoft/test/test.py
[2] https://lab.nexedi.com/nexedi/slapos/tree/15871bbf/software/ors-amarisoft/test

/cc @lu.xu, @tomo, @xavier_thompson, @Daetalus
/reviewed-by @jhuge
/reviewed-on !129

    slapos.toolbox$ rst2html.py CHANGES.txt CHANGES.html
    CHANGES.txt:33: (WARNING/2) Bullet list ends without a blank line; unexpected unindent.

-> Fix it.

Currently this promise is implemented by grepping whole rf_info output
for "HW" and "SW" strings. But this won't work ok in the presence of
multiple CPRI devices. Imagine, for example if one device has CPRI lock,
while the other does not:

    PCIe CPRI /dev/sdr2@1:
      Hardware ID: 0x4b12
      DNA: [0x0048248a334a7054]
      Serial: ''
      FPGA revision: 2023-06-23  10:05:24
      FPGA vccint: 0.98 V
      FPGA vccaux: 1.76 V
      FPGA vccbram: 0.98 V
      FPGA temperature: 71.9 °C
      Clock tune: 0.0 ppm
      NUMA: 0
      CPRI_option: '5' (x8) lock=no                     <-- NOTE
      DMA0: TX fifo: 66.67us  Usage=16/32768 (0%)
      DMA0: RX fifo: 66.67us  Usage=16/32768 (0%)
      DMA0 Underflows: 0
      DMA0 Overflows: 0
    PCIe CPRI /dev/sdr3@1:
      Hardware ID: 0x4b12
      DNA: [0x0048248a334a7054]
      Serial: ''
      FPGA revision: 2023-06-23  10:05:24
      FPGA vccint: 0.98 V
      FPGA vccaux: 1.77 V
      FPGA vccbram: 0.98 V
      FPGA temperature: 71.7 °C
      Clock tune: 0.0 ppm
      NUMA: 0
      CPRI_option: '5' (x8) lock=HW+SW rx/tx=46.606us   <-- NOTE
        Port #0: T14=46.606us
      DMA0: TX fifo: 66.67us  Usage=16/32768 (0%)
      DMA0: RX fifo: 66.67us  Usage=16/32768 (0%)
      DMA0 Underflows: 0
      DMA0 Overflows: 0

the old code would still report "CPRI locked all ok" and also globally
without indicating which CPRI channel is locked.

-> Fix it by adjusting check_cpri_lock to parse rf_info text more
precisely, detect devices there and to understand which device has CPRI
lock and which does not.

For now this change is accompanied by the following change in
ors-amarisoft SR to keep it working:

    --- a/software/ors-amarisoft/instance-enb.jinja2.cfg
    +++ b/software/ors-amarisoft/instance-enb.jinja2.cfg
    @@ -35,7 +35,6 @@ parts =
       check-lopcomm-sync.py
       check-lopcomm-config-log.py
       check-lopcomm-stats-log.py
    -  check-cpri-lock.py
     {% endif %}
     {% if slapparameter_dict.get("dnsmasq", None) %}
       dnsmasq-service
    @@ -48,6 +47,7 @@ parts =
     {% endif %}
       monitor-base
       publish-connection-information
    +{% set extra_part_list = [] %}

     extends = {{ monitor_template }}

    @@ -688,12 +688,21 @@ config-testing = {{ slapparameter_dict.get("testing", False) }}
     config-config-log = ${lopcomm-rrh-config-template:log-output}
     config-stats-period = {{ slapparameter_dict.get("enb_stats_fetch_period", 60) }}

    -[check-cpri-lock.py]
    +{%  if ru == "lopcomm" %}
    +{%-   set cell_list = slapparameter_dict.get('cell_list', {'default': {}}) %}
    +{%-   for i, k in enumerate(cell_list) %}
    +{%-     set sfp_port = cell_list[k].get('cpri_port_number', i) %}
    +{%-     do extra_part_list.append('SFP{{sfp_port}}-cpri-lock.py') %}
    +[SFP{{sfp_port}}-cpri-lock.py]
     <= macro.promise
     promise = check_cpri_lock
     config-testing = {{ slapparameter_dict.get("testing", False) }}
    +config-sdr_dev  = {{ slapparameter_dict.get('sdr_number', 0) }}
    +config-sfp_port = {{ sfp_port }}
     config-amarisoft-rf-info-log = ${amarisoft-rf-info-template:log-output}
     config-stats-period = {{ slapparameter_dict.get("enb_stats_fetch_period", 60) }}
    +{%-  endfor %}
    +{% endif %}

     [check-rx-saturated.py]
     <= macro.promise
    @@ -702,3 +711,9 @@ config-testing = {{ slapparameter_dict.get("testing", False) }}
     config-amarisoft-stats-log = ${amarisoft-stats-template:log-output}
     config-stats-period = {{ slapparameter_dict.get("enb_stats_fetch_period", 60) }}
     config-max-rx-sample-db = {{ slapparameter_dict.get("max_rx_sample_db", 0) }}
    +
    +[buildout]
    +parts +=
    +{%- for part in extra_part_list %}
    +    {{ part }}
    +{%- endfor %}

(posted in slapos!1461)

The way rf_info text is parsed could be also useful in the future to
e.g. detect FPGA revision of the boards and report their recency status
via promise.

/cc @jhuge, @tomo, @xavier_thompson, @Daetalus
/reviewed-by @lu.xu
/reviewed-on !127

This promise plugin was using tail_file(self.amarisoft_rf_info_log) to
get the most recent log entry to check. But tail_file simply return last
line of the log file without checking timestamp of returned entry. So it
could be the case that last line is ages ago, reports "CPRI locked ok"
and then, if there are no more rf_info entries for one reason or
another, the CPRI-locked promise will be holding green despite the fact
that data became stale.

-> Fix it by explicitly checking timestamp of last log entry to be in
expected recent range.

/cc @jhuge, @tomo, @xavier_thompson, @Daetalus
/reviewed-by @lu.xu
/reviewed-on !127

promise/plugin/check_cpri_lock: test: Factor common code to write to log and promise into common place

We are going to add more tests. Keeping the code to initialize test
environment duplicated over and over will be not convenient.

/cc @jhuge, @tomo, @xavier_thompson, @Daetalus
/reviewed-by @lu.xu
/reviewed-on !127