1. 18 Feb, 2024 1 commit
    • Kirill Smelkov's avatar
      software/ors-amarisoft: Do not recreate slaptapX-* on every idempotent `slapos node instance` run · 7989e6ce
      Kirill Smelkov authored
      To run tapsplit we use plone.recipe.command with both command and
      update-command set to `tapsplit ...`. But tapsplit, when run, currently fully
      recreates and reinitializes subtap interfaces, which leads to interfering with
      running enb because subtap interfaces, that enb started to use, are removed.
      This is not desirable behaviour.
      
      What we need:
      
      1) create subtap interfaces only once and keep them stable
      2) until configuration changes which should lead to
         * subtaps recreated, and
         * enb restarted
      3) if subtap interfaces disappear for any reason, recreate it
      
      -> Rework tapsplit to keep its promise, that it "brings tap interface into state
         with several children interfaces each covering part of original interface
         address space", without recreating those children on every run and instead
         doing any action only if their state is not what is desired.
      
      In other words those interfaces now are only created when they do not exist
      before. Addresses and routes are added only if they are not there before
      tapsplit is run, etc.
      
      After the patch the first run of tapsplit to split by 2 looks like
      
          # ./pythonwitheggs ru/tapsplit slaptap16 2
          slaptap16: split 2401:5180:0:66:a200::/71 by 2
          preserve         2401:5180:0:66:a200::/73
          -> slaptap16-1   2401:5180:0:66:a280::/73
           # ip tuntap add dev slaptap16-1 mode tap user slapuser16
           # ip link set slaptap16-1 up
           # ip addr add 2401:5180:0:66:a280::/73 dev slaptap16-1 noprefixroute
           # ip route add 2401:5180:0:66:a280::1 dev slaptap16-1
           # ip route add 2401:5180:0:66:a280::/73 dev slaptap16-1 via 2401:5180:0:66:a280::1
          -> slaptap16-2   2401:5180:0:66:a300::/73
           # ip tuntap add dev slaptap16-2 mode tap user slapuser16
           # ip link set slaptap16-2 up
           # ip addr add 2401:5180:0:66:a300::/73 dev slaptap16-2 noprefixroute
           # ip route add 2401:5180:0:66:a300::1 dev slaptap16-2
           # ip route add 2401:5180:0:66:a300::/73 dev slaptap16-2 via 2401:5180:0:66:a300::1
      
      The second run with the same arguments looks as
      
          # ./pythonwitheggs ru/tapsplit slaptap16 2
          slaptap16: split 2401:5180:0:66:a200::/71 by 2
          preserve         2401:5180:0:66:a200::/73
          -> slaptap16-1   2401:5180:0:66:a280::/73
           # slaptap16-1: already exists
           # slaptap16-1: already up
           # slaptap16-1: already has 2401:5180:0:66:a280::/73 addr
           # slaptap16-1: already has 2401:5180:0:66:a280::1 route
           # slaptap16-1: already has 2401:5180:0:66:a280::/73 route
          -> slaptap16-2   2401:5180:0:66:a300::/73
           # slaptap16-2: already exists
           # slaptap16-2: already up
           # slaptap16-2: already has 2401:5180:0:66:a300::/73 addr
           # slaptap16-2: already has 2401:5180:0:66:a300::1 route
           # slaptap16-2: already has 2401:5180:0:66:a300::/73 route
      
      where it could be seen that no actions had been taken.
      
      And if, for example, the user manipulates slaptap16-2 and manually sets it
      down, the third run restores it to desired 'UP' state and readds the address
      and routes because the kernel removed them when link went down:
      
          # ip -6 addr show dev slaptap16-2
          157: slaptap16-2: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc fq_codel state DOWN group default qlen 1000
              inet6 2401:5180:0:66:a300::/73 scope global tentative noprefixroute
                 valid_lft forever preferred_lft forever
          # ip -6 route show dev slaptap16-2
          2401:5180:0:66:a300::1 metric 1024 linkdown pref medium
          2401:5180:0:66:a300::/73 via 2401:5180:0:66:a300::1 metric 1024 linkdown pref medium
          # ip link set slaptap16-2 down
          # ip -6 addr show dev slaptap16-2
          # ip -6 route show dev slaptap16-2
          # ./pythonwitheggs ru/tapsplit slaptap16 2
          slaptap16: split 2401:5180:0:66:a200::/71 by 2
          preserve         2401:5180:0:66:a200::/73
          -> slaptap16-1   2401:5180:0:66:a280::/73
           # slaptap16-1: already exists
           # slaptap16-1: already up
           # slaptap16-1: already has 2401:5180:0:66:a280::/73 addr
           # slaptap16-1: already has 2401:5180:0:66:a280::1 route
           # slaptap16-1: already has 2401:5180:0:66:a280::/73 route
          -> slaptap16-2   2401:5180:0:66:a300::/73
           # slaptap16-2: already exists
           # ip link set slaptap16-2 up
           # ip addr add 2401:5180:0:66:a300::/73 dev slaptap16-2 noprefixroute
           # ip route add 2401:5180:0:66:a300::1 dev slaptap16-2
           # ip route add 2401:5180:0:66:a300::/73 dev slaptap16-2 via 2401:5180:0:66:a300::1
      
      The first version of this patch tried to solve the problem by setting
      update-command to be noop instead of reworking tapsplit itself. But as Thomas
      noted this does not satisfy requirement "3".
      
      Amends 49ce8ef5 (software/ors-amarisoft: Provide dedicated TAP interface for each Radio Unit)
      
      /helped-by @tomo
      /cc @jhuge, @lu.xu, @xavier_thompson, @Daetalus
      /reviewed-on nexedi/slapos!1508
      7989e6ce
  2. 16 Feb, 2024 11 commits
    • Jérome Perrin's avatar
      ERP5: Move frontend virtualhost logic on backend · 7ff43824
      Jérome Perrin authored
       - use caucase for balancer certificate
       - move virtual host logic on the backend
       - change "frontend" parameter to request "" type (and no longer "zope")
      
      See merge request nexedi/slapos!1504
      7ff43824
    • Jérome Perrin's avatar
      stack/erp5: implement Zope's rewrite rules in ERP5 balancer partition · 6e735808
      Jérome Perrin authored
      The strategy for compatibility is that:
       - haproxy still listen on the same port as before, without rewrite rule.
         This is called "legacy" port.
       - for each frontend from request parameters, we introduce an haproxy
         frontend with a rewrite for the corresponding `internal-path`
         parameter.
       - the shared frontend instance is updated to use this new frontend
         entry from haproxy. This will cause a small downtime until the shared
         frontend is updated to the new URL on ERP5, but since this feature
         was not used, it's OK.
      
      Technical details are that we:
       - split haproxy config to have frontends and backends.
       - introduce one frontend in haproxy for each frontend from request
         parameters.
       - routing-rule-list argument is still honored the same way, globally
         and after path from frontend.
       - change the shared frontend requests to use "" type, no longer "zope"
         type.
       - we don't do automatic detection of /VirtualHostRoot in URL but always
         add it, because it could be used to trick zope into thinking it
         serves requests for an arbitrary host and do open redirects
       - before using the request's host header in virtualhost path, we check
         that it does not contain /, to prevent injection of virutalhost path
         elements through the host header.
       - we don't use the "path" parameter from shared frontend, because we
         want the frontend to be simple, so we don't want it to rewrite the
         request path (which is also the reason why we deprecated "zope" type)
       - the tests have changed a lot, because they were using what's now the
         "legacy" URL types, so we updated it to use the new URL types with
         all the /VirtualHostRoot/../ in path and also because they use IPv6
         URL, no longer IPv4
      6e735808
    • Jérome Perrin's avatar
      5b3fc1f2
    • Jérome Perrin's avatar
      stack/erp5: use slapos.recipe.build to manage haproxy parameters · 2fc522bf
      Jérome Perrin authored
      and save the already allocated ports in a state file, so that requesting
      new families does not change already allocated ports.
      2fc522bf
    • Jérome Perrin's avatar
      stack/erp5: use caucase managed certificate for balancer · d49914a6
      Jérome Perrin authored
      This reverts commit 620c9332 (stack/erp5: stop using caucase managed
      certificate for balancer, 2020-11-10) with an updated design. We add a
      caucase service for balancer in the balancer partition. The caucase
      service from the root partition (that was not used) is removed.
      
      The underlying idea is that the default configuration should use multiple
      caucases with limited scope, here we have one caucase to manage the
      certificate used by haproxy server in the balancer partition, so we put
      one caucase to manage this certificate and the caucase is configured to
      auto-accept one certificate only. The plan is that when we will add a
      certificate for mariadb server, we'll add another caucase inside this
      mariadb server.
      
      For more advanced usage and also to support the cases where a new
      certificate needs to be re-emitted for some reason, users can request
      with an existing caucase URL. In that case, they will have to accept
      the certificate requests.
      
      Notable changes:
      
      balancer/ssl/caucase-url is no longer documented in parameters, this is
      an internal parameter, users can pass one global caucase service to
      manage all partition
      
      CAUCASE environment variable is no longer set when running zope. There
      was no identified use case and with this new approach of multiple
      caucases, the term "caucase" alone became ambiguous.
      d49914a6
    • Jérome Perrin's avatar
      stack/erp5: remove not used "backend-path" · 16c9df39
      Jérome Perrin authored
      This is not documented in schema and has no effect in erp5 (but this is
      still used for slapos-master)
      16c9df39
    • Jérome Perrin's avatar
    • Jérome Perrin's avatar
      ERP5: rework frontend instance parameter · cb78214e
      Jérome Perrin authored
      This change the format or the (mostly) unused frontend parameter to
      support requesting more than one frontend and also enable the request of
      a frontend by default, so that requesting a frontend separately is no
      longer needed.
      
      The `frontend` parameter now also supports requesting frontends for
      specific paths on the ERP5 backend, the example below requests a
      frontend serving directly a web site, with the necessary rewrite rules:
      
      ```js
      {
        "frontend": {
          "default": {
            "internal-path": "/erp5/web_site_module/renderjs_runner/"
          }
        }
      }
      ```
      
      The example below requests a default frontend to the erp5 root, to
      access the ZMI or erp5_xhtml_style interface and two web sites:
      
      ```js
      {
        "frontend": {
          "default": {},
          "erp5js": {
            "internal-path": "/erp5/web_site_module/renderjs_runner/"
          },
          "crm": {
            "internal-path": "/erp5/web_site_module/erp5_officejs_support_request_ui/"
          }
        }
      }
      ```
      
      The example below has an explicit definition of the zope families using
      `zope-partition-dict` parameter, because there is more than one zope
      family, no frontend is requested by default:
      
      ```js
      {
        "zope-partition-dict": {
          "backoffice": {
            "family": "backoffice"
          },
          "web": {
            "family": "web"
          },
          "activities": {
            "family": "activities"
          }
        }
      }
      ```
      
      Continuing this example, to have frontends for backoffice and web
      families, the frontend request can specify the families, like it is
      demonstrated in the example below. In this example, we don't specify an
      entry for "activities" family, so no frontend will be requested for
      this family.
      
      ```js
      {
        "frontend": {
          "backoffice": {
            "zope-family": "backoffice"
          },
          "web": {
            "zope-family": "web",
            "internal-path": "/erp5/web_site_module/web_site/"
          }
        }
        "zope-partition-dict": {
          "backoffice": {
            "family": "backoffice"
          },
          "web": {
            "family": "web"
          },
          "activities": {
            "family": "activities"
          }
        }
      }
      ```
      cb78214e
    • Jérome Perrin's avatar
      33d36fdb
    • Kirill Smelkov's avatar
      stack/slapos: v↑ slapos.toolbox to 0.142 · 73f44633
      Kirill Smelkov authored
      Going 0.140 -> 0.142 introduces the following changes:
      
          nexedi/slapos.toolbox@0.140...0.142
      
      Test results: https://erp5js.nexedi.net/#/test_result_module/20240215-D4F3CE96 (all ok)
      73f44633
    • Jérome Perrin's avatar
      software/metabase: version up v0.48.6 · 2a0ddd26
      Jérome Perrin authored
      2a0ddd26
  3. 15 Feb, 2024 1 commit
  4. 14 Feb, 2024 1 commit
  5. 13 Feb, 2024 2 commits
  6. 09 Feb, 2024 1 commit
  7. 08 Feb, 2024 1 commit
  8. 07 Feb, 2024 3 commits
  9. 05 Feb, 2024 2 commits
  10. 03 Feb, 2024 1 commit
  11. 01 Feb, 2024 4 commits
  12. 31 Jan, 2024 1 commit
  13. 30 Jan, 2024 11 commits
    • Kirill Smelkov's avatar
      software/ors-amarisoft: enb.jinja2.cfg: Increase SNR in handover code · 7c3c96e4
      Kirill Smelkov authored
      Don't repeat slapparameter_dict['ncell_list'][k] - we can introduce a name for
      current neighbour cell, and use that name in the loop. And align entries to
      that they read more clearly.
      
      No non-whitespace changes in rendered enb.cfg and gnb.cfg .
      
      /cc @jhuge, @lu.xu, @tomo, @xavier_thompson, @Daetalus
      /proposed-for-review-on nexedi/slapos!1526
      /reviewed-by TrustMe
      7c3c96e4
    • Kirill Smelkov's avatar
      software/ors-amarisoft: enb.jinja2.cfg: Factor handover code into one place · 41d23ce2
      Kirill Smelkov authored
      We currently have LTE-specific handover configuration in under cell_list and
      NR-specific handover configuration in under nr_cell_list. Those configuration
      are different.
      
      However in upcoming MultiRU we will need to handle LTE->NR handover, NR->LTE
      handover, and also Intra-ENB handover in addition to Inter-ENB handover we
      currently do.
      
      -> Move handover code into common function as a preparatory step for that.
      
      In the future the handover code for LTE and NR cells will be the same, so it
      makes sense to move that code to common place to avoid duplication.
      
      For rendered enb.cfg this unification introduces only space and trivial
      changes as shown in the appendix.
      
      /cc @jhuge, @lu.xu, @tomo, @xavier_thompson, @Daetalus
      /proposed-for-review-on nexedi/slapos!1528
      /reviewed-by TrustMe
      
      Appendix. Diff for rendered enb.cfg and gnb.cfg before and after this patch
      
      ```
      $ git diff -w --no-index config/{old,out}
      ```
      
      ```diff
      diff --git a/config/old/enb.cfg b/config/out/enb.cfg
      index 43301ee13..9dcca16c7 100644
      --- a/config/old/enb.cfg
      +++ b/config/out/enb.cfg
      @@ -45,16 +45,18 @@
             root_sequence_index: 204,
             dl_earfcn: 36100,
             inactivity_timer: 10000,
      +
             // Handover
             ncell_list: [
               // Inter-ENB HO
               {
      +          rat: "eutra",
                 n_id_cell: 35,
                 dl_earfcn: 700,
                 cell_id: 0x12345,
                 tac: 123,
      -        }],
      -
      +        },
      +      ],
      
             // Carrier Aggregation
             scell_list: [
      diff --git a/config/old/gnb.cfg b/config/out/gnb.cfg
      index 2127a2f6b..23b07d6e1 100644
      --- a/config/old/gnb.cfg
      +++ b/config/out/gnb.cfg
      @@ -57,6 +57,7 @@
           ssb_pos_bitmap: "10000000",
      
           inactivity_timer: 10000,
      +
             // Handover
             ncell_list: [
               // Inter-ENB HO
      @@ -74,8 +75,8 @@
                 ssb_period: 20,
                 ssb_offset: 0,
                 ssb_duration: 1,
      -      }],
      -
      +        },
      +      ],
      
           // tune NR parameters for the cell
           manual_ref_signal_power: true,
      ```
      41d23ce2
    • Kirill Smelkov's avatar
      software/ors-amarisoft: slapos-render-config: Test handover · 3fd11a89
      Kirill Smelkov authored
      Useful to have while doing handover-related changes.
      
      /cc @jhuge, @lu.xu, @tomo, @xavier_thompson, @Daetalus
      /proposed-for-review-on nexedi/slapos!1528
      /reviewed-by TrustMe
      
      Appendix. Diff for rendered enb.cfg and gnb.cfg before and after this patch
      
      ```
      $ git diff --no-index config/{old,out}
      ```
      
      ```diff
      diff --git a/config/old/enb.cfg b/config/out/enb.cfg
      index 1843e0f24..43301ee13 100644
      --- a/config/old/enb.cfg
      +++ b/config/out/enb.cfg
      @@ -45,6 +45,16 @@
             root_sequence_index: 204,
             dl_earfcn: 36100,
             inactivity_timer: 10000,
      +      // Handover
      +      ncell_list: [
      +        // Inter-ENB HO
      +        {
      +          n_id_cell: 35,
      +          dl_earfcn: 700,
      +          cell_id: 0x12345,
      +          tac: 123,
      +        }],
      +
      
             // Carrier Aggregation
             scell_list: [
      diff --git a/config/old/gnb.cfg b/config/out/gnb.cfg
      index d76b45d3c..2127a2f6b 100644
      --- a/config/old/gnb.cfg
      +++ b/config/out/gnb.cfg
      @@ -57,6 +57,25 @@
           ssb_pos_bitmap: "10000000",
      
           inactivity_timer: 10000,
      +    // Handover
      +    ncell_list: [
      +      // Inter-ENB HO
      +      {
      +        rat: "nr",
      +        dl_nr_arfcn: 520000,
      +        ssb_nr_arfcn: ,
      +        ul_nr_arfcn: 520000,
      +        n_id_cell: 75,
      +        gnb_id_bits: 22,
      +        nr_cell_id: 0x77712,
      +        tac: 321,
      +        band: 38,
      +        ssb_subcarrier_spacing: 30,
      +        ssb_period: 20,
      +        ssb_offset: 0,
      +        ssb_duration: 1,
      +      }],
      +
      ```
      3fd11a89
    • Kirill Smelkov's avatar
      software/ors-amarisoft: enb.jinja2.cfg: Unify logging · 04323c6d
      Kirill Smelkov authored
      1. Currently we have separate log_options for LTE and NR cases with listing
      different subsystems in each. But in MultiRU one enb will be driving both LTE
      and NR cells at the same time, so we will need to define both LTE- and
      NR-related levels.
      
      -> Merge all log settings into one log_options as a preparatory step
      
      For current state it does not hurt for an LTE if we set e.g. ngap.level, and it
      does not hurt for NR if we set e.g. s1ap.level - since those layers will be
      unused. This way merging log settings for both LTE and NR subsystems is ok.
      
      --------
      
      2. Factorize log_phy_debug handling: instead of duplicating whole
      log_options line and changing only phy.level settings there, construct the
      log_options line programmatically and handle phy.level on its own.
      
      --------
      
      3. Use log/enb.log log_filename for both LTE and NR cases. In the upcoming
      MultiRU there might be several cells activated at the same time and in general
      it will be not possible to say are we doing "enb" or "gnb" now - for example if
      there will be two cells - one LTE and one NR.
      
      -> Use enb.log for log filename uniformly similarly to how the software is
      named (lteenb) even though it can work as both enb and gnb.
      
      For the reference we do the same with enb.xlog in nexedi/slapos!1522 .
      
      /cc @jhuge, @lu.xu, @tomo, @xavier_thompson, @Daetalus
      /proposed-for-review-on nexedi/slapos!1527
      /reviewed-by TrustMe
      
      Appendix. Diff for rendered enb.cfg and gnb.cfg before and after this patch
      
      ```
      $ git diff --no-index config/{old,out}
      ```
      
      ```diff
      diff --git a/config/old/enb.cfg b/config/out/enb.cfg
      index 467bb6364..1843e0f24 100644
      --- a/config/old/enb.cfg
      +++ b/config/out/enb.cfg
      @@ -3,9 +3,7 @@
      
       {
      -
      -  log_options: "all.level=error,all.max_size=0,nas.level=debug,nas.max_size=1,s1ap.level=debug,s1ap.max_size=1,x2ap.level=debug,x2ap.max_size=1,rrc.level=debug,rrc.max_size=1,phy.level=info,file.rotate=1G,file.path=/dev/null",
      -
      +  log_options: "all.level=error,all.max_size=0,nas.level=debug,nas.max_size=1,s1ap.level=debug,s1ap.max_size=1,x2ap.level=debug,x2ap.max_size=1,rrc.level=debug,rrc.max_size=1,ngap.level=debug,ngap.max_size=1,xnap.level=debug,xnap.max_size=1,phy.level=info,file.rotate=1G,file.path=/dev/null",
         log_filename: "log/enb.log",
      
      diff --git a/config/old/gnb.cfg b/config/out/gnb.cfg
      index 18523818a..d76b45d3c 100644
      --- a/config/old/gnb.cfg
      +++ b/config/out/gnb.cfg
      @@ -3,10 +3,8 @@
      
       {
      -
      -  log_options: "all.level=error,all.max_size=0,nas.level=debug,nas.max_size=1,ngap.level=debug,ngap.max_size=1,xnap.level=debug,xnap.max_size=1,rrc.level=debug,rrc.max_size=1,phy.level=info,file.rotate=1G,file.path=/dev/null",
      -
      -  log_filename: "log/gnb.log",
      +  log_options: "all.level=error,all.max_size=0,nas.level=debug,nas.max_size=1,s1ap.level=debug,s1ap.max_size=1,x2ap.level=debug,x2ap.max_size=1,rrc.level=debug,rrc.max_size=1,ngap.level=debug,ngap.max_size=1,xnap.level=debug,xnap.max_size=1,phy.level=info,file.rotate=1G,file.path=/dev/null",
      +  log_filename: "log/enb.log",
      
         rf_driver: {
      ```
      04323c6d
    • Kirill Smelkov's avatar
      software/ors-amarisoft: test: Preprocess enb.cfg & co on YAML loading · d8b0a558
      Kirill Smelkov authored
      Current enb config is already quite complex and with MultiRU it will be growing
      more - both with added features and with more sections emitted because there
      will be multiple radio units, multiple cells and cross cell interactions. So
      for clarity we will want to annotate with a comment to which cell or ru object
      a section belongs, or to which cell-cell pair a particular interaction belongs.
      
      Amarisoft supports C-style comments and preprocessor directives out of the box,
      but if we use them in the configuration files, yaml.load, that we use in the
      test to load generated configs, will break, because // and /* ... */ is not
      valid YAML. It looks like Amarisoft does preprocessing as a separate step
      before further loading given configuration via yaml.
      
      So to be able to use the comments and still have tests working we need to do
      the same - in the tests preprocess the files before feeding them to yaml loader.
      
      -> Do that with the help of https://pypi.org/project/pcpp/
      
      In my view that library has good quality and in my experience it worked
      flawlessly. Anyway we need it to only handle comments, not sophisticated CPP
      features, and for that it works just ok.
      
      Add some comments to existing enb.cfg and ue.cfg to make sure it really works.
      Those are simple comments, and in current state it they might seem as not 100%
      necessary, but with more upcoming config changes it would be good to have those
      descriptionary anchors present in the generated configs, so I suggest we add them.
      Anyway they should not do any harm at all.
      
      /cc @jhuge, @lu.xu, @tomo, @xavier_thompson, @Daetalus
      /proposed-for-review-on nexedi/slapos!1526
      /reviewed-by TrustMe
      d8b0a558
    • Kirill Smelkov's avatar
      software/ors-amarisoft: test: Avoid YAMLLoadWarning warning · 82d3852c
      Kirill Smelkov authored
      Currently on every test yaml.load issues a warning that using it without
      explicitly specifying loader is deprecated, for example:
      
          test_enb_conf (testTDD.TestENBParameters) .../slapos/software/ors-amarisoft/test/test.py:29: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
      
      -> Fix the warning by using appropriate loader explicitly.
      
      Use FullLoader since msg.pyyaml.org/load describes it as
      
         Loads the full YAML language. Avoids arbitrary code execution.
         This is currently (PyYAML 5.1+) the default loader called by yaml.load(input) (after issuing the warning).
      
      i.e. we get support for full YAML feature set, do not get code execution and
      make sure that the behaviour stays exactly the same as before, but without the
      warning.
      
      /cc @jhuge, @lu.xu, @tomo, @xavier_thompson, @Daetalus
      /proposed-for-review-on nexedi/slapos!1526
      /reviewed-by TrustMe
      82d3852c
    • Kirill Smelkov's avatar
      software/ors-amarisoft: test: Factor loading YAML config into dedicated function · ff40ed40
      Kirill Smelkov authored
      We are going to do fixes and enhancements to YAML loading processes. Move
      corresponding code to one place before that as the preparatory step.
      
      Place new utility in new test/test.py file - with MultiRU that will be the
      place for tests of generic mode and test.jinja2.py, which will be renamed to
      test_ors.py, will be using utilities from there. So place the utility into new
      place from the start.
      
      /cc @jhuge, @lu.xu, @tomo, @xavier_thompson, @Daetalus
      /proposed-for-review-on !1526
      /reviewed-by TrustMe
      ff40ed40
    • Kirill Smelkov's avatar
      software/ors-amarisoft: Merge instance-gnb.jinja2.cfg into instance-enb.jinja2.cfg · 503e50a4
      Kirill Smelkov authored
      In MultiRU one enb will be driving multiple, possible of different kind,
      TDD/FDD and LTE/NR cells all at the same time, because that standard
      functionality of a base station and because original Amarisoft software
      supports that out of the box.
      
      However we currently have enb and gnb services separate and duplicating most of
      each other code.
      
      -> Merge instance-enb.jinja2.cfg and instance-gnb.jinja2.cfg as a preparatory step for MultiRU.
      
      - instance-enb.jinja2.cfg pulls gnb specific bits from instance-gnb.jinja2.cfg
      - instance-gnb.jinja2.cfg goes away effectively switching gnb to use
        ru/libinstance.jinja2.cfg like enb already does
      - For parameters that were duplicated as enb_<X> and gnb_<X>  generic software
        now accepts only enb_<X> with ORS mode wrapper providing backward
        compatibility for gnb_* parameters. For example if gnb_config_link is given
        for a gnb instance, it will be translated to enb_config_link to underlying
        generic enb so that gnb_config_link works correctly.
      
        For ORS we care to provide 100% backward compatibility because there are many ORS'es deployed.
        For generic software we are free to clean things up as needed, because there
        is not much generic deployments at this time.
      
      /cc @jhuge, @lu.xu, @tomo, @xavier_thompson, @Daetalus
      /proposed-for-review-on nexedi/slapos!1522
      /reviewed-by TrustMe
      
      --------
      
      Appendix. Diff between instance-enb.jinja2.cfg and instance-gnb.jinja2.cfg before this patch
      
      ```
      $ git diff --no-index instance-enb.jinja2.cfg instance-gnb.jinja2.cfg
      ```
      
      ```diff
      diff --git a/instance-enb.jinja2.cfg b/instance-gnb.jinja2.cfg
      index 6d4ce7e94..6ee62b3ff 100644
      --- a/instance-enb.jinja2.cfg
      +++ b/instance-gnb.jinja2.cfg
      @@ -1,19 +1,25 @@
       {#- defaults for eNB radio parameters.
           NOTE they are installed temporary and will go away after switch to generic multiRU. #}
       {%- do RF.setdefault('tx_gain',     slapparameter_dict.get('tx_gain',     0)) %}
       {%- do RF.setdefault('rx_gain',     slapparameter_dict.get('rx_gain',     0)) %}
      -{%- do RF.setdefault('dl_earfcn',   slapparameter_dict.get('dl_earfcn',   0)) %}
      +{%- do RF.setdefault('dl_nr_arfcn', slapparameter_dict.get('dl_nr_arfcn', 0)) %}
      +{%- do RF.setdefault('nr_band',     slapparameter_dict.get('nr_band',     0)) %}
      
       [buildout]
       parts =
         directory
      -  enb-config
      +  gnb-config
         enb-service
         xamari-xlog-service
       {% if slapparameter_dict.get('xlog_fluentbit_forward_host') %}
         xlog-fluentbit-service
       {% endif %}
      +  amarisoft-stats-service
      +  amarisoft-rf-info-service
      +  check-sdr-busy.py
         check-baseband-latency.py
      +  check-amarisoft-stats-log.py
      +  check-rx-saturated.py
         monitor-base
         publish-connection-information
      
      @@ -23,11 +29,6 @@ eggs-directory = {{ eggs_directory }}
       develop-eggs-directory = {{ develop_eggs_directory }}
       offline = true
      
      -{%- import 'slaplte.jinja2'            as slaplte with context  %}
      -{%- import 'ru_libinstance.jinja2.cfg' as rulib   with context  %}
      -{{ rulib.buildout() }}
      -
      -
       [monitor-httpd-conf-parameter]
       httpd-include-file = {{ buildout_directory }}/etc/httpd-include-file.conf
       port = ${monitor-instance-parameter:monitor-httpd-port}
      @@ -52,10 +53,11 @@ cert = {{ slap_connection['cert-file'] }}
      
       configuration.com_ws_port = 9001
       configuration.com_addr = 127.0.1.2
      -configuration.mme_addr = 127.0.1.100
      +configuration.amf_addr = 127.0.1.100
       configuration.gtp_addr = 127.0.1.1
      -configuration.default_lte_bandwidth = {{ default_lte_bandwidth }}
      -configuration.default_lte_inactivity_timer = {{ default_lte_inactivity_timer }}
      +configuration.default_nr_bandwidth = {{ default_nr_bandwidth }}
      +configuration.default_nr_inactivity_timer = {{ default_nr_inactivity_timer }}
      +configuration.default_nr_ssb_pos_bitmap = {{ default_nr_ssb_pos_bitmap }}
       configuration.default_n_antenna_dl = {{ default_n_antenna_dl }}
       configuration.default_n_antenna_ul = {{ default_n_antenna_ul }}
      
      @@ -63,6 +65,7 @@ configuration.default_n_antenna_ul = {{ default_n_antenna_ul }}
       recipe = slapos.cookbook:mkdirectory
       software = {{ buildout_directory }}
       home = ${buildout:directory}
      +etc = ${:home}/etc
       var = ${:home}/var
       etc = ${:home}/etc
       bin = ${:home}/bin
      @@ -73,30 +76,32 @@ service = ${:etc}/service
       promise = ${:etc}/promise
       log = ${:var}/log
      
      -{% if slapparameter_dict.get("enb_config_link", None) %}
      -[enb-config-dl]
      +{% if slapparameter_dict.get("gnb_config_link", None) %}
      +[gnb-config-dl]
       recipe = slapos.recipe.build:download
      -url = {{ slapparameter_dict.get("enb_config_link") }}
      -version = {{ slapparameter_dict.get("enb_config_version") }}
      +url = {{ slapparameter_dict.get("gnb_config_link") }}
      +version = {{ slapparameter_dict.get("gnb_config_version") }}
       offline = false
       {% endif %}
      
       [enb-sh-wrapper]
       recipe = slapos.recipe.template
       output = ${directory:bin}/${:_buildout_section_name_}
      -enb-log = ${directory:log}/enb-output.log
      +gnb-log = ${directory:log}/gnb-output.log
       inline =
         #!/bin/sh
       {% if not slapparameter_dict.get("testing", False) %}
         sudo -n /opt/amarisoft/rm-tmp-lte;
         sudo -n /opt/amarisoft/init-sdr;
         sudo -n /opt/amarisoft/init-enb;
      -  (echo && echo && date "+[%Y/%m/%d %T.%N %Z] Starting eNB software..." && echo) >> ${:enb-log};
      -  tail -c 1M ${:enb-log} > ${:enb-log}.tmp;
      -  mv ${:enb-log}.tmp ${:enb-log};
      -  {{ enb }}/lteenb ${directory:etc}/enb.cfg >> ${:enb-log} 2>> ${:enb-log};
      +  (echo && echo && date "+[%Y/%m/%d %T.%N %Z] Starting gNB software..." && echo) >> ${:gnb-log};
      +  tail -c 1M ${:gnb-log} > ${:gnb-log}.tmp;
      +  mv ${:gnb-log}.tmp ${:gnb-log};
      +  {{ enb }}/lteenb ${directory:etc}/gnb.cfg >> ${:gnb-log} 2>> ${:gnb-log};
      +
       {% endif %}
      
      +### eNodeB (enb)
       [enb-service]
       recipe = slapos.cookbook:wrapper
       command-line = ${enb-sh-wrapper:output}
      @@ -105,7 +110,7 @@ mode = 0775
       reserve-cpu = True
       pidfile = ${directory:run}/enb.pid
       hash-files =
      -  ${enb-config:output}
      +  ${gnb-config:output}
         ${enb-sh-wrapper:output}
       environment =
         LD_LIBRARY_PATH={{ openssl_location }}/lib
      @@ -114,22 +119,23 @@ environment =
       [xamari-xlog-script]
       recipe = slapos.recipe.template
       output = ${directory:bin}/${:_buildout_section_name_}
      -period = {{ slapparameter_dict.get("enb_stats_fetch_period", 60) }}
      +period = {{ slapparameter_dict.get("gnb_stats_fetch_period", 60) }}
       stats_logspec = stats[samples,rf]/${:period}s
      -{%- if slapparameter_dict.get("enb_drb_stats_enabled", True) %}
      +{%- if slapparameter_dict.get("gnb_drb_stats_enabled", True) %}
       drb_stats_logspec = x.drb_stats/${:period}s
       {%- else %}
       drb_stats_logspec =
       {%- endif %}
       rotatespec = 100MB.9
       logspec = ${:stats_logspec} ${:drb_stats_logspec}
      -{%- if slapparameter_dict.get("websocket_password", "") %}
      +logspec = ${:stats_logspec} ${:drb_stats_logspec}
      +{% if slapparameter_dict.get("websocket_password", "") %}
       websock = ws://[${slap-configuration:ipv6-random}]:9001
      -{%- else %}
      +{% else %}
       websock = ws://127.0.1.2:9001
      -{%- endif %}
      +{% endif %}
       xamari = {{ buildout_directory }}/bin/xamari
      -logfile = ${monitor-directory:public}/enb.xlog
      +logfile = ${monitor-directory:public}/gnb.xlog
       inline =
         #!/bin/sh
         exec ${:xamari} xlog --rotate ${:rotatespec} ${:websock} ${:logfile} ${:logspec}
      @@ -177,6 +183,52 @@ wrapper-path = ${directory:service}/${:_buildout_section_name_}
       hash-files = ${:fluentbit-config}
       {% endif %}
      
      +[amarisoft-stats-template]
      +recipe = slapos.recipe.template:jinja2
      +extensions = jinja2.ext.do
      +log-output = ${directory:var}/log/amarisoft-stats.json.log
      +context =
      +  section directory directory
      +  key slapparameter_dict slap-configuration:configuration
      +  key log_file :log-output
      +  raw stats_period {{ slapparameter_dict.get("gnb_stats_fetch_period", 60) }}
      +  raw testing {{ slapparameter_dict.get("testing", False) }}
      +  raw python_path {{ buildout_directory}}/bin/pythonwitheggs
      +mode = 0775
      +url = {{ ru_amarisoft_stats_template }}
      +output = ${directory:bin}/amarisoft-stats.py
      +
      +[amarisoft-rf-info-template]
      +recipe = slapos.recipe.template:jinja2
      +extensions = jinja2.ext.do
      +log-output = ${directory:var}/log/amarisoft-rf-info.json.log
      +context =
      +  section directory directory
      +  key slapparameter_dict slap-configuration:configuration
      +  key log_file :log-output
      +  raw stats_period {{ slapparameter_dict.get("gnb_stats_fetch_period", 60) }}
      +  raw testing {{ slapparameter_dict.get("testing", False) }}
      +  raw python_path {{ buildout_directory}}/bin/pythonwitheggs
      +mode = 0775
      +url = {{ ru_amarisoft_rf_info_template }}
      +output = ${directory:bin}/amarisoft-rf-info.py
      +
      +[amarisoft-stats-service]
      +recipe = slapos.cookbook:wrapper
      +command-line = ${amarisoft-stats-template:output}
      +wrapper-path = ${directory:service}/amarisoft-stats
      +mode = 0775
      +hash-files =
      +  ${amarisoft-stats-template:output}
      +
      +[amarisoft-rf-info-service]
      +recipe = slapos.cookbook:wrapper
      +command-line = ${amarisoft-rf-info-template:output}
      +wrapper-path = ${directory:service}/amarisoft-rf-info
      +mode = 0775
      +hash-files =
      +  ${amarisoft-rf-info-template:output}
      +
       [config-base]
       recipe = slapos.recipe.template:jinja2
       extensions = jinja2.ext.do
      @@ -190,53 +242,47 @@ context =
         raw gtp_addr_v4 {{ lan_ipv4 }}
         raw tx_gain {{ RF.tx_gain }}
         raw rx_gain {{ RF.rx_gain }}
      -  raw earfcn  {{ RF.dl_earfcn }}
      +  raw nr_arfcn {{ RF.dl_nr_arfcn }}
      +  raw nr_band {{ RF.nr_band }}
         raw software_name {{ software_name }}
         raw rf_mode {{ rf_mode }}
         raw trx {{ trx }}
         raw bbu {{ bbu }}
         raw ru_type {{ ru }}
      -  json do_lte true
      -  json do_nr  false
      +  json do_lte false
      +  json do_nr  true
         import  netaddr netaddr
         ${:extra-context}
      
      -[sib-config]
      -<= config-base
      -url = {{ sib23_template }}
      -output = ${directory:etc}/sib23.cfg
      -
       [drb-config]
       <= config-base
      -url = {{ drb_lte_template }}
      +url = {{ drb_nr_template }}
       output = ${directory:etc}/drb.cfg
      
      -[enb-config]
      +[gnb-config]
       <= config-base
      -{% if slapparameter_dict.get("enb_config_link", None) %}
      -url = ${enb-config-dl:target}
      +{% if slapparameter_dict.get("gnb_config_link", None) %}
      +url = ${gnb-config-dl:target}
       {% else %}
       url = {{ enb_template }}
       {% endif %}
      -output = ${directory:etc}/enb.cfg
      +output = ${directory:etc}/gnb.cfg
       extra-context =
           import json_module json
      -    json cell_list {{ rulib.cell_list | tojson }}
      -    key sib23_file sib-config:output
           key drb_file   drb-config:output
       import-list =
           rawfile slaplte.jinja2 {{ slaplte_template }}
      
      -
       [publish-connection-information]
       <= monitor-publish
       recipe = slapos.cookbook:publish.serialised
       {%- if slapparameter_dict.get("websocket_password", "") %}
       websocket_url = ws://[${slap-configuration:ipv6-random}]:9001
       {%- endif %}
      -enb-ipv6 = ${slap-configuration:ipv6-random}
      -enb-ipv4 = {{ lan_ipv4 }}
      -current-earfcn  = {{ RF.dl_earfcn }}
      +gnb-ipv6 = ${slap-configuration:ipv6-random}
      +gnb-ipv4 = {{ lan_ipv4 }}
      +current-nr-arfcn = {{ RF.dl_nr_arfcn }}
      +current-nr-band = {{ RF.nr_band }}
       amarisoft-version = {{ lte_version }}
       license-expiration = {{ lte_expiration }}
       monitor-gadget-url = ${:monitor-base-url}/gadget/software.cfg.html
      @@ -253,10 +299,35 @@ password = {{ slapparameter_dict['monitor-password'] | string }}
       <= monitor-promise-base
       name = ${:_buildout_section_name_}
      
      +[check-sdr-busy.py]
      +<= macro.promise
      +promise = check_sdr_busy
      +config-testing = {{ slapparameter_dict.get("testing", False) }}
      +config-sdr = {{ sdr }}
      +config-sdr_dev  = 0
      +config-dma_chan = 0
      +
       [check-baseband-latency.py]
       <= macro.promise
       promise = check_baseband_latency
       config-testing = {{ slapparameter_dict.get("testing", False) }}
      -config-amarisoft-stats-log = ${ru_amarisoft-stats-template:log-output}
      -config-stats-period = {{ slapparameter_dict.get("enb_stats_fetch_period", 60) }}
      +config-amarisoft-stats-log = ${amarisoft-stats-template:log-output}
      +config-stats-period = {{ slapparameter_dict.get("gnb_stats_fetch_period", 60) }}
       config-min-rxtx-delay = {{ slapparameter_dict.get("min_rxtx_delay", 0) }}
      +
      +[check-amarisoft-stats-log.py]
      +<= macro.promise
      +promise = check_amarisoft_stats_log
      +output = ${directory:plugins}/check-amarisoft-stats-log.py
      +config-testing = {{ slapparameter_dict.get("testing", False) }}
      +config-amarisoft-stats-log = ${amarisoft-stats-template:log-output}
      +config-stats-period = {{ slapparameter_dict.get("gnb_stats_fetch_period", 60) }}
      +
      +[check-rx-saturated.py]
      +<= macro.promise
      +promise = check_rx_saturated
      +config-testing = {{ slapparameter_dict.get("testing", False) }}
      +config-rf-rx-chan-list = {{ list(range(0, int(slapparameter_dict.get('n_antenna_ul', default_n_antenna_ul)))) }}
      +config-amarisoft-stats-log = ${amarisoft-stats-template:log-output}
      +config-stats-period = {{ slapparameter_dict.get("gnb_stats_fetch_period", 60) }}
      +config-max-rx-sample-db = {{ slapparameter_dict.get("max_rx_sample_db", 0) }}
      ```
      503e50a4
    • Kirill Smelkov's avatar
      software/ors-amarisoft: Start to introduce ORS mode · f3f1cb46
      Kirill Smelkov authored
      In MultiRU we are going to have generic software with flexible configuration
      and many features + ORS software, which wraps the generic part, adds ORS
      specific features, and translates simple ORS schema to generic ones.
      
      Let's first move ORS-specific bits to dedicated place as a preparatory step for that:
      
      - add software-ors.cfg . In the future software-ors.cfg will be the entry point
        for ORS mode, but for now it is only a place from where software-ors-* extend from.
      
      - add instance-ors.cfg . We move ORS-specific code from instance.cfg here and
        set it to use instance-ors-enb.jinja2.cfg when instantiating enb/gnb.
      
      - add instance-ors-enb.jinja2.cfg . This pulls ORS-specific code from
        instance-enb.jinja2.cfg + instance-gnb.jinja2.cfg and wraps them with ORS
        context built by instance-ors.cfg
      
      - in the templates the way to see whether it is ORS or not, and if yes, to access
        ORS parameters is changed: now it is always via checking `ors` object, which
        can be either False for non-ORS mode, or dict for ORS mode, and if it is dict
        the keys inside that dict are ORS specific properties, e.g. ors['one-watt'].
      
        This adjustment is handy to organize ORS mode extension from generic with
        generic setting `json ors false` in the context, and ORS mode undoing that and
        setting ORS context properly. I think the code also reads more clear this way, i.e. with
      
       	{%- if one_watt == "True" %}
      
        changed to
      
      	{%- if ors['one-watt'] %}
      
        because in the second variant it is clear that one-watt is a property of ORS.
      
      The rendered configs for enb.cfg and gnb.cfg stay unchanged.
      
      /cc @jhuge, @lu.xu, @tomo, @xavier_thompson, @Daetalus
      /proposed-for-review-on !1522
      /reviewed-by TrustMe
      f3f1cb46
    • Kirill Smelkov's avatar
      software/ors-amarisoft: Rename ors-id -> comp-id · a6a4d2d4
      Kirill Smelkov authored
      We use computer identifier in fluentd forwarding. This identifier was named as
      ors-id, but it applies not only to ORS, but also to generic deployment with
      BBU.
      
      In the next patch we are going to introduce ORS mode and move ORS-specific
      bits to software-ors.cfg and instance-ors.cfg + co. The computer identifier
      will stay in generic part, because it is generic.
      
      -> To avoid ambiguity rename it to be clearly unrelated to ORS specifics.
      
      /cc @jhuge, @lu.xu, @tomo, @xavier_thompson, @Daetalus
      /proposed-for-review-on nexedi/slapos!1522
      /reviewed-by TrustMe
      a6a4d2d4
    • Kirill Smelkov's avatar
      software/ors-amarisoft: enb.jinja2.cfg: Tune PUCCH 1 and PUCCH 3 for CA · eab4a41b
      Kirill Smelkov authored
      When there are multiple cells and CA configured, without those parameters enb fails to start with e.g.
      
          n1_pucch_an_cs_count must be > 0 for the CA Primary cell
      
      Set them on all cell like illustrated in enb-2cc.cfg and enb-3cc.cfg Amarisoft examples.
      
      Not sure selected values are best - I took them from above mentioned examples as is.
      
      Reference to documentation about those parameters:
      
      https://tech-academy.amarisoft.com/lteenb.doc#prop.pucch_dedicated.n1_pucch_an_cs_count
      https://tech-academy.amarisoft.com/lteenb.doc#prop.pucch_dedicated.n3_pucch_an_n_rb
      https://tech-academy.amarisoft.com/lteenb.doc#prop.pucch_dedicated.ack_nack_feedback_mode_ca
      
      /cc @lu.xu, @tomo, @xavier_thompson, @Daetalus
      /helped-by @jhuge
      /reviewed-on nexedi/slapos!1519
      eab4a41b