1. 12 Feb, 2024 13 commits
    • Kirill Smelkov's avatar
      software/ors-amarisoft: enb: Push Lopcomm-specific parameters to be per-RU · f16af895
      Kirill Smelkov authored
      Like e.g. txa0cc00_center_frequency, rxa0cc00_center_frequency_earfcn, etc -
      because different units needs to use different parameters.
      
      cron_schedule is renamed to -> reset_schedule to better reflect its meaning.
      
      user-authorized-key remains to be global as this is per-eNB nor per-RU
      property, but is likely unneded in the future. Corresponding FIXME is added to
      highlight this.
      
      Backward compatibility: no change for ORS; lopcomm parameters become per-cell and need adaptation.
      f16af895
    • Kirill Smelkov's avatar
      software/ors-amarisoft: enb: Start to generalize the code to handle multiple Radio Units · 0fc23eca
      Kirill Smelkov authored
      Add code to organize a registry of Radio Units and handle that registry generally everywhere.
      
      RU registry is still populated from cell_list and in practice there still can
      be radio units only of the same type, but besides slaplte.load_ru_and_cell
      which is aware of that, the rest of the code tries to do everything as if
      RUs of different type could be present simultaneously.
      
      Now it is not only lopcomm who can do multiCELL, but also sunwave and SDR as well.
      
      gNB also starts to gain support for cell_list, because cell_list loading is
      uniformly applied to both eNB and gNB. However, since enb.cfg is not yet
      prepared to handle multiple NR cells yet, there is an assert that in case of NR
      there is only one RU/cell present there. We will remove this limitation in a
      follow-up patch.
      
      Later we will also change the loading to load RU descriptions from shared instances
      and they won't be constrained to be Radio Units of the same type. But we need
      to prepare a lot to be able to do that.
      
      One more step forward towards MultiRU.
      
      Tests will be added later as full tests for generic MultiRU.
      
      Backward compatibility: no change for ORS and practically no breaking change for everything else.
      
      --------
      
      Appendix. Diff for rendered enb.cfg and gnb.cfg before and after this patch:
      
      ```
      $ ./pythonwitheggs slapos-render-config.py && git diff -w --no-index config/{old,out}
      ```
      
      ```diff
      diff --git a/config/old/enb.cfg b/config/out/enb.cfg
      index 884483b0a..cafdf42be 100644
      --- a/config/old/enb.cfg
      +++ b/config/out/enb.cfg
      @@ -1,24 +1,22 @@
      
      -
       {
         log_options: "all.level=error,all.max_size=0,nas.level=debug,nas.max_size=1,s1ap.level=debug,s1ap.max_size=1,x2ap.level=debug,x2ap.max_size=1,rrc.level=debug,rrc.max_size=1,ngap.level=debug,ngap.max_size=1,xnap.level=debug,xnap.max_size=1,phy.level=info,file.rotate=1G,file.path=/dev/null",
         log_filename: "log/enb.log",
      
      -
      +  // Radio Units
         rf_driver: {
      +      // default-RU 2T2R  (ors)
             name: "sdr",
             args: "dev0=/dev/sdr0",
      -
             rx_antenna:"tx_rx",
             tdd_tx_mod: 1,
         },
      -  tx_gain: 62,
      -  rx_gain: 43,
      -
      +  tx_gain: [62, 62],
      +  rx_gain: [43, 43],
         com_addr: "127.0.1.2:9001",
         // LTE core network
         mme_list: [
      @@ -36,6 +34,8 @@
      
         // LTE cells
         cell_list: [
      +
      +    // default  (default-RU)
           {
             rf_port:      0,
             n_antenna_dl: 2,
      diff --git a/config/old/gnb.cfg b/config/out/gnb.cfg
      index fd57ca3dc..7818b4ea8 100644
      --- a/config/old/gnb.cfg
      +++ b/config/out/gnb.cfg
      @@ -1,24 +1,22 @@
      
      -
       {
         log_options: "all.level=error,all.max_size=0,nas.level=debug,nas.max_size=1,s1ap.level=debug,s1ap.max_size=1,x2ap.level=debug,x2ap.max_size=1,rrc.level=debug,rrc.max_size=1,ngap.level=debug,ngap.max_size=1,xnap.level=debug,xnap.max_size=1,phy.level=info,file.rotate=1G,file.path=/dev/null",
         log_filename: "log/enb.log",
      
      -
      +  // Radio Units
         rf_driver: {
      +      // default-RU 2T2R  (ors)
             name: "sdr",
             args: "dev0=/dev/sdr0",
      -
             rx_antenna:"tx_rx",
             tdd_tx_mod: 1,
         },
      -  tx_gain: 62,
      -  rx_gain: 43,
      -
      +  tx_gain: [62, 62],
      +  rx_gain: [43, 43],
         com_addr: "127.0.1.2:9001",
         // NR core network
         amf_list: [
      ```
      0fc23eca
    • Kirill Smelkov's avatar
      software/ors-amarisoft: enb: lte: Fix tdd_ul_dl_config default in schema · 35303462
      Kirill Smelkov authored
      For LTE tdd_ul_dl_config is
      
          enum [
               "[Configuration 2] 5ms 2UL 6DL (default)",
               "[Configuration 6] 5ms 5UL 3DL (maximum uplink)"
          ],
      
      but default was outside of that enum. The code in enb.jinja2.cfg actually uses
      the first option from the enum as the default.
      
      -> Fix default in schema.
      
      tdd_ul_dl_config in NR seems to be already correct.
      35303462
    • Kirill Smelkov's avatar
      software/ors-amarisoft: enb: cpri_port parameter is unused · c4b2f4dc
      Kirill Smelkov authored
      There is cell_list.*.cpri_port_number that is used insted from the beginning.
      
      -> Remove unused cpri_port from scheamas.
      c4b2f4dc
    • Kirill Smelkov's avatar
      software/ors-amarisoft: ue: Switch UE type to be runtime parameter · 53641314
      Kirill Smelkov authored
      Previously type of emulated UE was static parameter of particular software
      release - it was possible to simulate UE and attach to cells only of the RAT of
      particular template.
      
      In MultiRU it will be possible to generally emulate all kind of UEs - both LTE
      and NR all at the same time, and attach them to LTE and NR cells
      simultaneously.
      
      -> Switch type of UE to be runtime parameter as a preparatory step for that.
      
      URLs of software releases changes (we remove lte and nr in names), but here we
      do not care about backward compatibility because currently there are just a few
      UEsim deployments and migration should be easy.
      53641314
    • Kirill Smelkov's avatar
      software/ors-amarisoft: ue: Reuse SIM object in UE schemas · c3de8831
      Kirill Smelkov authored
      Less code duplication
      c3de8831
    • Kirill Smelkov's avatar
      software/ors-amarisoft: Move JSON schema of SIM card to sim/ · 07c27e40
      Kirill Smelkov authored
      SIM card object is useful to have not only for Core Network - we will also use
      it for UE configutation. Move SIM schema into common place as a preparatory step.
      07c27e40
    • Kirill Smelkov's avatar
      software/ors-amarisoft: ue: Make default K and IMSI to be common for LTE and NR · 6f321938
      Kirill Smelkov authored
      K and IMSI are orthogonal to RAT and, similarly to how we keep default
      n_antenna_dl/ul independent of RAT we can do so with this default UE parameters.
      6f321938
    • Kirill Smelkov's avatar
      software/ors-amarisoft: ue: Don't run UEsim tests in ORS mode · 865a466b
      Kirill Smelkov authored
      ORS does not support UEsim as indicated by `if bbu != ors` in software.cfg.json.jinja2.
      
      Amends f3f1cb46 (software/ors-amarisoft: Start to introduce ORS mode) because
      that patch started to really reject instantiating ue-* on ORS.
      865a466b
    • Kirill Smelkov's avatar
      software/ors-amarisoft: ue: Don't render UEsim input schemas for ORS · de76bf0a
      Kirill Smelkov authored
      Those were unused because software.cfg.json.jinja2 explicitly enables UEsim only for bbu != 'ors'.
      de76bf0a
    • Kirill Smelkov's avatar
      software/ors-amarisoft: render-templates: Rework --delete to be precise · 2c207449
      Kirill Smelkov authored
      Soon we are going to make rf_mode a runtime parameter and so there
      won't be TDD and FDD in software names anymore.
      
      Prepare to that and make --delete to remove the exact set of files that
      render-template actually generates instead of becoming a noop after "runtime
      rf_mode" restructuring.
      2c207449
    • Kirill Smelkov's avatar
      software/ors-amarisoft: enb+ue: Query slapparameter_dict from master only once · 785d0f56
      Kirill Smelkov authored
      We are currently querying it in instance.cfg and later, once again, in e.g. instance-enb.jinja2.cfg .
      
      However ORS mode will need to adjust slapparameter_dict with ORS defaults
      and pass it further to generic enb. So it won't work if we use
      slapparameter_dict obtained the second time because the second query will
      return unadjusted original slapparameter_dict.
      
      In this patch we are only doing preparatory step - redo the code not query the
      master from instance-{enb,ue}.jinja2.cfg and work with slapparameter_dict and
      slap_configuration already queried by instance.cfg
      
      For UE the change is trivial.
      
      For eNB instance-enb.jinja2.cfg used to set the defaults for com, mme, amf and
      gtp in the same section used for the second query. We rework those defaults to
      be applied to slapparameter_dict via jinja2 - via the same way we are going to
      later use in ORS mode.
      
      Handling defaults for everything besides gtp_addr is straightforward. For
      gtp_addr it has the semantic difference when explicitly given, and given only
      implicitly. In the latter case the intent of original code is to autodetect
      whether to use an address on loopback, or externally-visible address.
      
      Original code used the check for emptiness of mme_list/amf_list as the
      condition for "use loopback". Since now those lists, after applying their
      defaults, are never empty we rework the code to see if core address is on the
      loopback or not, and use auto-GTP-on-loopback only if core is also on loopback.
      This should, hopefully, be more convenient as it also works ok out of the box
      if core is on loopback, but its address was explicitly specified. Previously
      for such cases gtp_addr was also needed to be specified, and now it should work
      without that.
      
      No change to rendered enb.cfg and gnb.cfg besides whitespace.
      
      Adjust ipv6-random usage in core-network for consistency with enb as well.
      785d0f56
    • Kirill Smelkov's avatar
      software/ors-amarisoft: enb: dnsmasq-service :port and :ip are unused and misleading · 64008abb
      Kirill Smelkov authored
      For example it is the port that is setup in ru/dnsmasq.jinja2.cfg that is used,
      not :port and there it is 5354, i.e. different. :ip is also not used anywhere.
      
      -> Remove them to avoid confusion.
      64008abb
  2. 07 Feb, 2024 8 commits
    • Kirill Smelkov's avatar
      software/ors-amarisoft: enb: Rename cell_list -> cell_dict internally · 35a0c77e
      Kirill Smelkov authored
      That data structure is really a dict, not list. It is more clear to name it appropriately.
      
      -> Do the renaming but keep cell_list intact as described in JSON schemas so
         that external interface remains unchanged.
      
      Rendered enb.cfg and gnb.cfg remain the same.
      35a0c77e
    • Kirill Smelkov's avatar
      software/ors-amarisoft: enb+ue: Make ru_type and do_lte/do_nr to be accessible at instance level · c4c74310
      Kirill Smelkov authored
      Not only inside rendered enb.jinja2.cfg and ue.jinja2.cfg because we will need
      that information in slaplte and slaplte is imported by instance-enb and
      instance-ue.
      c4c74310
    • Kirill Smelkov's avatar
      software/ors-amarisoft: enb+ue: Stop using slap_configuration to propagate LTE and NR defaults · a65d6fa2
      Kirill Smelkov authored
      instance-enb and instance-ue currently setup that slap_configuration with
      LTE/NR defaults to be accessible from enb.jinja2.cfg and ue.jinja2.cfg. But we
      will soon need to have access to those defaults from slaplte.jinja2 as well,
      and it will break if left as is because when slaplte is imported from e.g.
      enb.jinja2 - it will work, but when slaplte is imported from instance-enb it
      will break because parent of instance-enb (instance.cfg) does not setup
      defaults in slap_configuration at that level.
      
      The fix is to either duplicate slap_configuration at instance.cfg level, or to
      switch access to the defaults to go via original default_* parameters.
      
      We go the second way for simplicity.
      a65d6fa2
    • Kirill Smelkov's avatar
      software/ors-amarisoft: enb.jinja2.cfg: Switch internal ru.ru_type to be... · c5b76d54
      Kirill Smelkov authored
      software/ors-amarisoft: enb.jinja2.cfg: Switch internal ru.ru_type to be 'sunwave' for Sunwave M2RU Radio Unit
      
      ru/libinstance and everything inside there already refer to that unit as 'sunwave'.
      Do the same for uniformity in enb.cfg and because we already refer to Lopcomm
      ORAN Radio Unit as just 'lopcomm'.
      
      In the future, if we will need to distinguish different models of one
      manufacturer, we could extend ru type to be e.g. manufacturer/model or do
      something similar.
      
      Template rendering is still done with ru='m2ru' coming from outside but
      internally it is now ru.ru_type='sunwave' instead of ru.ru_type='m2ru'.
      c5b76d54
    • Kirill Smelkov's avatar
      software/ors-amarisoft: enb.jinja2.cfg: Inline cell.bandwidth for now · 85e31d7b
      Kirill Smelkov authored
      Even though I introduced cell object in 79370ebf (software/ors-amarisoft:
      enb.jinja2.cfg: Stop using C Preprocessor and switch to Jinja2 completely) it
      is currently only cell.bandwidth that is living there, while all other cell
      parameters are still fetched from slapparameter_dict directly.
      
      We will soon introduce cell variable - that will iterate over cell_dict, and
      also keeping global cell won't work due to name shadowing. On the other hand we
      are not yet ready to start migrating all cell parameters to be accessible via
      cell object.
      
      -> So inline code for lte bandwidth parameter until the time comes to rework
      all cell parameters to be accessible via cell object.
      
      Rendered enb.cfg and gnb.cfg stay the same.
      85e31d7b
    • Kirill Smelkov's avatar
      software/ors-amarisoft: Move rf_config code from enb.jinja2.cfg -> to slaplte.jinja2 · ca3958da
      Kirill Smelkov authored
      To handle multiple radio units we will need to rework this code significantly.
      Move this to a dedicated routine as a preparatory step. The code moves out of
      enb.cfg because later it will be also used in UEsim to configure simulator
      radio units as well.
      
      No non-whitespace changes in rendered enb.cfg and gnb.cfg .
      
      /cc @jhuge, @lu.xu, @tomo, @xavier_thompson, @Daetalus
      ca3958da
    • Kirill Smelkov's avatar
      software/ors-amarisoft: Do not recreate slaptapX-* on every idempotent `slapos node instance` run · 3ffab700
      Kirill Smelkov authored
      To run tapsplit we use plone.recipe.command with both command and
      update-command set to `tapsplit ...`. But tapsplit, when run, fully
      recreates and reinitializes subtap interfaces, which leads to
      interfering with running enb because subtap interfaces, that enb
      started to use, are removed. This is not desirable behaviour.
      
      What we need:
      
      1) create subtap interfaces only once and keep them stable
      2) until configuration changes which should lead to
         * subtaps recreated, and
         * enb restarted
      
      Carefully reading plone.recipe.command documentation shows:
      
        command
          Command to run when the buildout part is installed.
      
        update-command
          Command to run when the buildout part is updated. This happens when
          buildout is run BUT THE CONFIGURATION FOR THIS BUILDOUT PART HAS NOT
          CHANGED.
      
        (emphasis mine)
      
      So the fix looks to be to make update-command noop - this fulfills
      requirement "1". For "2" - I've verified that when configuration
      changes, e.g. number of RU changes, buildout reinstalls [vtap] section
      from scratch, and it also should restart enb, because generated enb.cfg
      changes.
      
      So the fix is fully correct because it satifies all needed requirements.
      
      Amends 49ce8ef5 (software/ors-amarisoft: Provide dedicated TAP interface for each Radio Unit)
      /cc @jhuge, @lu.xu, @tomo, @xavier_thompson, @Daetalus
      3ffab700
    • Kirill Smelkov's avatar
      fixup! software/ors-amarisoft: Start to introduce ORS mode · 346abf72
      Kirill Smelkov authored
      In f3f1cb46 I've made a mistake in "inject ORS defaults" code - there it should
      be RF.dl_earfcn and RF.dl_nr_arfcn instead of RF.earfcn and RF.nr_arfcn.
      346abf72
  3. 30 Jan, 2024 13 commits
    • Kirill Smelkov's avatar
      X ktesting · 62e4182d
      Kirill Smelkov authored
      Test only ors-amarisoft and from kirr/slapos@knext + kirr/slapos.toolbox@next .
      62e4182d
    • Kirill Smelkov's avatar
      software/ors-amarisoft: enb.jinja2.cfg: Increase SNR in handover code · 7c3c96e4
      Kirill Smelkov authored
      Don't repeat slapparameter_dict['ncell_list'][k] - we can introduce a name for
      current neighbour cell, and use that name in the loop. And align entries to
      that they read more clearly.
      
      No non-whitespace changes in rendered enb.cfg and gnb.cfg .
      
      /cc @jhuge, @lu.xu, @tomo, @xavier_thompson, @Daetalus
      /proposed-for-review-on nexedi/slapos!1526
      /reviewed-by TrustMe
      7c3c96e4
    • Kirill Smelkov's avatar
      software/ors-amarisoft: enb.jinja2.cfg: Factor handover code into one place · 41d23ce2
      Kirill Smelkov authored
      We currently have LTE-specific handover configuration in under cell_list and
      NR-specific handover configuration in under nr_cell_list. Those configuration
      are different.
      
      However in upcoming MultiRU we will need to handle LTE->NR handover, NR->LTE
      handover, and also Intra-ENB handover in addition to Inter-ENB handover we
      currently do.
      
      -> Move handover code into common function as a preparatory step for that.
      
      In the future the handover code for LTE and NR cells will be the same, so it
      makes sense to move that code to common place to avoid duplication.
      
      For rendered enb.cfg this unification introduces only space and trivial
      changes as shown in the appendix.
      
      /cc @jhuge, @lu.xu, @tomo, @xavier_thompson, @Daetalus
      /proposed-for-review-on nexedi/slapos!1528
      /reviewed-by TrustMe
      
      Appendix. Diff for rendered enb.cfg and gnb.cfg before and after this patch
      
      ```
      $ git diff -w --no-index config/{old,out}
      ```
      
      ```diff
      diff --git a/config/old/enb.cfg b/config/out/enb.cfg
      index 43301ee13..9dcca16c7 100644
      --- a/config/old/enb.cfg
      +++ b/config/out/enb.cfg
      @@ -45,16 +45,18 @@
             root_sequence_index: 204,
             dl_earfcn: 36100,
             inactivity_timer: 10000,
      +
             // Handover
             ncell_list: [
               // Inter-ENB HO
               {
      +          rat: "eutra",
                 n_id_cell: 35,
                 dl_earfcn: 700,
                 cell_id: 0x12345,
                 tac: 123,
      -        }],
      -
      +        },
      +      ],
      
             // Carrier Aggregation
             scell_list: [
      diff --git a/config/old/gnb.cfg b/config/out/gnb.cfg
      index 2127a2f6b..23b07d6e1 100644
      --- a/config/old/gnb.cfg
      +++ b/config/out/gnb.cfg
      @@ -57,6 +57,7 @@
           ssb_pos_bitmap: "10000000",
      
           inactivity_timer: 10000,
      +
             // Handover
             ncell_list: [
               // Inter-ENB HO
      @@ -74,8 +75,8 @@
                 ssb_period: 20,
                 ssb_offset: 0,
                 ssb_duration: 1,
      -      }],
      -
      +        },
      +      ],
      
           // tune NR parameters for the cell
           manual_ref_signal_power: true,
      ```
      41d23ce2
    • Kirill Smelkov's avatar
      software/ors-amarisoft: slapos-render-config: Test handover · 3fd11a89
      Kirill Smelkov authored
      Useful to have while doing handover-related changes.
      
      /cc @jhuge, @lu.xu, @tomo, @xavier_thompson, @Daetalus
      /proposed-for-review-on nexedi/slapos!1528
      /reviewed-by TrustMe
      
      Appendix. Diff for rendered enb.cfg and gnb.cfg before and after this patch
      
      ```
      $ git diff --no-index config/{old,out}
      ```
      
      ```diff
      diff --git a/config/old/enb.cfg b/config/out/enb.cfg
      index 1843e0f24..43301ee13 100644
      --- a/config/old/enb.cfg
      +++ b/config/out/enb.cfg
      @@ -45,6 +45,16 @@
             root_sequence_index: 204,
             dl_earfcn: 36100,
             inactivity_timer: 10000,
      +      // Handover
      +      ncell_list: [
      +        // Inter-ENB HO
      +        {
      +          n_id_cell: 35,
      +          dl_earfcn: 700,
      +          cell_id: 0x12345,
      +          tac: 123,
      +        }],
      +
      
             // Carrier Aggregation
             scell_list: [
      diff --git a/config/old/gnb.cfg b/config/out/gnb.cfg
      index d76b45d3c..2127a2f6b 100644
      --- a/config/old/gnb.cfg
      +++ b/config/out/gnb.cfg
      @@ -57,6 +57,25 @@
           ssb_pos_bitmap: "10000000",
      
           inactivity_timer: 10000,
      +    // Handover
      +    ncell_list: [
      +      // Inter-ENB HO
      +      {
      +        rat: "nr",
      +        dl_nr_arfcn: 520000,
      +        ssb_nr_arfcn: ,
      +        ul_nr_arfcn: 520000,
      +        n_id_cell: 75,
      +        gnb_id_bits: 22,
      +        nr_cell_id: 0x77712,
      +        tac: 321,
      +        band: 38,
      +        ssb_subcarrier_spacing: 30,
      +        ssb_period: 20,
      +        ssb_offset: 0,
      +        ssb_duration: 1,
      +      }],
      +
      ```
      3fd11a89
    • Kirill Smelkov's avatar
      software/ors-amarisoft: enb.jinja2.cfg: Unify logging · 04323c6d
      Kirill Smelkov authored
      1. Currently we have separate log_options for LTE and NR cases with listing
      different subsystems in each. But in MultiRU one enb will be driving both LTE
      and NR cells at the same time, so we will need to define both LTE- and
      NR-related levels.
      
      -> Merge all log settings into one log_options as a preparatory step
      
      For current state it does not hurt for an LTE if we set e.g. ngap.level, and it
      does not hurt for NR if we set e.g. s1ap.level - since those layers will be
      unused. This way merging log settings for both LTE and NR subsystems is ok.
      
      --------
      
      2. Factorize log_phy_debug handling: instead of duplicating whole
      log_options line and changing only phy.level settings there, construct the
      log_options line programmatically and handle phy.level on its own.
      
      --------
      
      3. Use log/enb.log log_filename for both LTE and NR cases. In the upcoming
      MultiRU there might be several cells activated at the same time and in general
      it will be not possible to say are we doing "enb" or "gnb" now - for example if
      there will be two cells - one LTE and one NR.
      
      -> Use enb.log for log filename uniformly similarly to how the software is
      named (lteenb) even though it can work as both enb and gnb.
      
      For the reference we do the same with enb.xlog in nexedi/slapos!1522 .
      
      /cc @jhuge, @lu.xu, @tomo, @xavier_thompson, @Daetalus
      /proposed-for-review-on nexedi/slapos!1527
      /reviewed-by TrustMe
      
      Appendix. Diff for rendered enb.cfg and gnb.cfg before and after this patch
      
      ```
      $ git diff --no-index config/{old,out}
      ```
      
      ```diff
      diff --git a/config/old/enb.cfg b/config/out/enb.cfg
      index 467bb6364..1843e0f24 100644
      --- a/config/old/enb.cfg
      +++ b/config/out/enb.cfg
      @@ -3,9 +3,7 @@
      
       {
      -
      -  log_options: "all.level=error,all.max_size=0,nas.level=debug,nas.max_size=1,s1ap.level=debug,s1ap.max_size=1,x2ap.level=debug,x2ap.max_size=1,rrc.level=debug,rrc.max_size=1,phy.level=info,file.rotate=1G,file.path=/dev/null",
      -
      +  log_options: "all.level=error,all.max_size=0,nas.level=debug,nas.max_size=1,s1ap.level=debug,s1ap.max_size=1,x2ap.level=debug,x2ap.max_size=1,rrc.level=debug,rrc.max_size=1,ngap.level=debug,ngap.max_size=1,xnap.level=debug,xnap.max_size=1,phy.level=info,file.rotate=1G,file.path=/dev/null",
         log_filename: "log/enb.log",
      
      diff --git a/config/old/gnb.cfg b/config/out/gnb.cfg
      index 18523818a..d76b45d3c 100644
      --- a/config/old/gnb.cfg
      +++ b/config/out/gnb.cfg
      @@ -3,10 +3,8 @@
      
       {
      -
      -  log_options: "all.level=error,all.max_size=0,nas.level=debug,nas.max_size=1,ngap.level=debug,ngap.max_size=1,xnap.level=debug,xnap.max_size=1,rrc.level=debug,rrc.max_size=1,phy.level=info,file.rotate=1G,file.path=/dev/null",
      -
      -  log_filename: "log/gnb.log",
      +  log_options: "all.level=error,all.max_size=0,nas.level=debug,nas.max_size=1,s1ap.level=debug,s1ap.max_size=1,x2ap.level=debug,x2ap.max_size=1,rrc.level=debug,rrc.max_size=1,ngap.level=debug,ngap.max_size=1,xnap.level=debug,xnap.max_size=1,phy.level=info,file.rotate=1G,file.path=/dev/null",
      +  log_filename: "log/enb.log",
      
         rf_driver: {
      ```
      04323c6d
    • Kirill Smelkov's avatar
      software/ors-amarisoft: test: Preprocess enb.cfg & co on YAML loading · d8b0a558
      Kirill Smelkov authored
      Current enb config is already quite complex and with MultiRU it will be growing
      more - both with added features and with more sections emitted because there
      will be multiple radio units, multiple cells and cross cell interactions. So
      for clarity we will want to annotate with a comment to which cell or ru object
      a section belongs, or to which cell-cell pair a particular interaction belongs.
      
      Amarisoft supports C-style comments and preprocessor directives out of the box,
      but if we use them in the configuration files, yaml.load, that we use in the
      test to load generated configs, will break, because // and /* ... */ is not
      valid YAML. It looks like Amarisoft does preprocessing as a separate step
      before further loading given configuration via yaml.
      
      So to be able to use the comments and still have tests working we need to do
      the same - in the tests preprocess the files before feeding them to yaml loader.
      
      -> Do that with the help of https://pypi.org/project/pcpp/
      
      In my view that library has good quality and in my experience it worked
      flawlessly. Anyway we need it to only handle comments, not sophisticated CPP
      features, and for that it works just ok.
      
      Add some comments to existing enb.cfg and ue.cfg to make sure it really works.
      Those are simple comments, and in current state it they might seem as not 100%
      necessary, but with more upcoming config changes it would be good to have those
      descriptionary anchors present in the generated configs, so I suggest we add them.
      Anyway they should not do any harm at all.
      
      /cc @jhuge, @lu.xu, @tomo, @xavier_thompson, @Daetalus
      /proposed-for-review-on nexedi/slapos!1526
      /reviewed-by TrustMe
      d8b0a558
    • Kirill Smelkov's avatar
      software/ors-amarisoft: test: Avoid YAMLLoadWarning warning · 82d3852c
      Kirill Smelkov authored
      Currently on every test yaml.load issues a warning that using it without
      explicitly specifying loader is deprecated, for example:
      
          test_enb_conf (testTDD.TestENBParameters) .../slapos/software/ors-amarisoft/test/test.py:29: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
      
      -> Fix the warning by using appropriate loader explicitly.
      
      Use FullLoader since msg.pyyaml.org/load describes it as
      
         Loads the full YAML language. Avoids arbitrary code execution.
         This is currently (PyYAML 5.1+) the default loader called by yaml.load(input) (after issuing the warning).
      
      i.e. we get support for full YAML feature set, do not get code execution and
      make sure that the behaviour stays exactly the same as before, but without the
      warning.
      
      /cc @jhuge, @lu.xu, @tomo, @xavier_thompson, @Daetalus
      /proposed-for-review-on nexedi/slapos!1526
      /reviewed-by TrustMe
      82d3852c
    • Kirill Smelkov's avatar
      software/ors-amarisoft: test: Factor loading YAML config into dedicated function · ff40ed40
      Kirill Smelkov authored
      We are going to do fixes and enhancements to YAML loading processes. Move
      corresponding code to one place before that as the preparatory step.
      
      Place new utility in new test/test.py file - with MultiRU that will be the
      place for tests of generic mode and test.jinja2.py, which will be renamed to
      test_ors.py, will be using utilities from there. So place the utility into new
      place from the start.
      
      /cc @jhuge, @lu.xu, @tomo, @xavier_thompson, @Daetalus
      /proposed-for-review-on nexedi/slapos!1526
      /reviewed-by TrustMe
      ff40ed40
    • Kirill Smelkov's avatar
      software/ors-amarisoft: Merge instance-gnb.jinja2.cfg into instance-enb.jinja2.cfg · 503e50a4
      Kirill Smelkov authored
      In MultiRU one enb will be driving multiple, possible of different kind,
      TDD/FDD and LTE/NR cells all at the same time, because that standard
      functionality of a base station and because original Amarisoft software
      supports that out of the box.
      
      However we currently have enb and gnb services separate and duplicating most of
      each other code.
      
      -> Merge instance-enb.jinja2.cfg and instance-gnb.jinja2.cfg as a preparatory step for MultiRU.
      
      - instance-enb.jinja2.cfg pulls gnb specific bits from instance-gnb.jinja2.cfg
      - instance-gnb.jinja2.cfg goes away effectively switching gnb to use
        ru/libinstance.jinja2.cfg like enb already does
      - For parameters that were duplicated as enb_<X> and gnb_<X>  generic software
        now accepts only enb_<X> with ORS mode wrapper providing backward
        compatibility for gnb_* parameters. For example if gnb_config_link is given
        for a gnb instance, it will be translated to enb_config_link to underlying
        generic enb so that gnb_config_link works correctly.
      
        For ORS we care to provide 100% backward compatibility because there are many ORS'es deployed.
        For generic software we are free to clean things up as needed, because there
        is not much generic deployments at this time.
      
      /cc @jhuge, @lu.xu, @tomo, @xavier_thompson, @Daetalus
      /proposed-for-review-on nexedi/slapos!1522
      /reviewed-by TrustMe
      
      --------
      
      Appendix. Diff between instance-enb.jinja2.cfg and instance-gnb.jinja2.cfg before this patch
      
      ```
      $ git diff --no-index instance-enb.jinja2.cfg instance-gnb.jinja2.cfg
      ```
      
      ```diff
      diff --git a/instance-enb.jinja2.cfg b/instance-gnb.jinja2.cfg
      index 6d4ce7e94..6ee62b3ff 100644
      --- a/instance-enb.jinja2.cfg
      +++ b/instance-gnb.jinja2.cfg
      @@ -1,19 +1,25 @@
       {#- defaults for eNB radio parameters.
           NOTE they are installed temporary and will go away after switch to generic multiRU. #}
       {%- do RF.setdefault('tx_gain',     slapparameter_dict.get('tx_gain',     0)) %}
       {%- do RF.setdefault('rx_gain',     slapparameter_dict.get('rx_gain',     0)) %}
      -{%- do RF.setdefault('dl_earfcn',   slapparameter_dict.get('dl_earfcn',   0)) %}
      +{%- do RF.setdefault('dl_nr_arfcn', slapparameter_dict.get('dl_nr_arfcn', 0)) %}
      +{%- do RF.setdefault('nr_band',     slapparameter_dict.get('nr_band',     0)) %}
      
       [buildout]
       parts =
         directory
      -  enb-config
      +  gnb-config
         enb-service
         xamari-xlog-service
       {% if slapparameter_dict.get('xlog_fluentbit_forward_host') %}
         xlog-fluentbit-service
       {% endif %}
      +  amarisoft-stats-service
      +  amarisoft-rf-info-service
      +  check-sdr-busy.py
         check-baseband-latency.py
      +  check-amarisoft-stats-log.py
      +  check-rx-saturated.py
         monitor-base
         publish-connection-information
      
      @@ -23,11 +29,6 @@ eggs-directory = {{ eggs_directory }}
       develop-eggs-directory = {{ develop_eggs_directory }}
       offline = true
      
      -{%- import 'slaplte.jinja2'            as slaplte with context  %}
      -{%- import 'ru_libinstance.jinja2.cfg' as rulib   with context  %}
      -{{ rulib.buildout() }}
      -
      -
       [monitor-httpd-conf-parameter]
       httpd-include-file = {{ buildout_directory }}/etc/httpd-include-file.conf
       port = ${monitor-instance-parameter:monitor-httpd-port}
      @@ -52,10 +53,11 @@ cert = {{ slap_connection['cert-file'] }}
      
       configuration.com_ws_port = 9001
       configuration.com_addr = 127.0.1.2
      -configuration.mme_addr = 127.0.1.100
      +configuration.amf_addr = 127.0.1.100
       configuration.gtp_addr = 127.0.1.1
      -configuration.default_lte_bandwidth = {{ default_lte_bandwidth }}
      -configuration.default_lte_inactivity_timer = {{ default_lte_inactivity_timer }}
      +configuration.default_nr_bandwidth = {{ default_nr_bandwidth }}
      +configuration.default_nr_inactivity_timer = {{ default_nr_inactivity_timer }}
      +configuration.default_nr_ssb_pos_bitmap = {{ default_nr_ssb_pos_bitmap }}
       configuration.default_n_antenna_dl = {{ default_n_antenna_dl }}
       configuration.default_n_antenna_ul = {{ default_n_antenna_ul }}
      
      @@ -63,6 +65,7 @@ configuration.default_n_antenna_ul = {{ default_n_antenna_ul }}
       recipe = slapos.cookbook:mkdirectory
       software = {{ buildout_directory }}
       home = ${buildout:directory}
      +etc = ${:home}/etc
       var = ${:home}/var
       etc = ${:home}/etc
       bin = ${:home}/bin
      @@ -73,30 +76,32 @@ service = ${:etc}/service
       promise = ${:etc}/promise
       log = ${:var}/log
      
      -{% if slapparameter_dict.get("enb_config_link", None) %}
      -[enb-config-dl]
      +{% if slapparameter_dict.get("gnb_config_link", None) %}
      +[gnb-config-dl]
       recipe = slapos.recipe.build:download
      -url = {{ slapparameter_dict.get("enb_config_link") }}
      -version = {{ slapparameter_dict.get("enb_config_version") }}
      +url = {{ slapparameter_dict.get("gnb_config_link") }}
      +version = {{ slapparameter_dict.get("gnb_config_version") }}
       offline = false
       {% endif %}
      
       [enb-sh-wrapper]
       recipe = slapos.recipe.template
       output = ${directory:bin}/${:_buildout_section_name_}
      -enb-log = ${directory:log}/enb-output.log
      +gnb-log = ${directory:log}/gnb-output.log
       inline =
         #!/bin/sh
       {% if not slapparameter_dict.get("testing", False) %}
         sudo -n /opt/amarisoft/rm-tmp-lte;
         sudo -n /opt/amarisoft/init-sdr;
         sudo -n /opt/amarisoft/init-enb;
      -  (echo && echo && date "+[%Y/%m/%d %T.%N %Z] Starting eNB software..." && echo) >> ${:enb-log};
      -  tail -c 1M ${:enb-log} > ${:enb-log}.tmp;
      -  mv ${:enb-log}.tmp ${:enb-log};
      -  {{ enb }}/lteenb ${directory:etc}/enb.cfg >> ${:enb-log} 2>> ${:enb-log};
      +  (echo && echo && date "+[%Y/%m/%d %T.%N %Z] Starting gNB software..." && echo) >> ${:gnb-log};
      +  tail -c 1M ${:gnb-log} > ${:gnb-log}.tmp;
      +  mv ${:gnb-log}.tmp ${:gnb-log};
      +  {{ enb }}/lteenb ${directory:etc}/gnb.cfg >> ${:gnb-log} 2>> ${:gnb-log};
      +
       {% endif %}
      
      +### eNodeB (enb)
       [enb-service]
       recipe = slapos.cookbook:wrapper
       command-line = ${enb-sh-wrapper:output}
      @@ -105,7 +110,7 @@ mode = 0775
       reserve-cpu = True
       pidfile = ${directory:run}/enb.pid
       hash-files =
      -  ${enb-config:output}
      +  ${gnb-config:output}
         ${enb-sh-wrapper:output}
       environment =
         LD_LIBRARY_PATH={{ openssl_location }}/lib
      @@ -114,22 +119,23 @@ environment =
       [xamari-xlog-script]
       recipe = slapos.recipe.template
       output = ${directory:bin}/${:_buildout_section_name_}
      -period = {{ slapparameter_dict.get("enb_stats_fetch_period", 60) }}
      +period = {{ slapparameter_dict.get("gnb_stats_fetch_period", 60) }}
       stats_logspec = stats[samples,rf]/${:period}s
      -{%- if slapparameter_dict.get("enb_drb_stats_enabled", True) %}
      +{%- if slapparameter_dict.get("gnb_drb_stats_enabled", True) %}
       drb_stats_logspec = x.drb_stats/${:period}s
       {%- else %}
       drb_stats_logspec =
       {%- endif %}
       rotatespec = 100MB.9
       logspec = ${:stats_logspec} ${:drb_stats_logspec}
      -{%- if slapparameter_dict.get("websocket_password", "") %}
      +logspec = ${:stats_logspec} ${:drb_stats_logspec}
      +{% if slapparameter_dict.get("websocket_password", "") %}
       websock = ws://[${slap-configuration:ipv6-random}]:9001
      -{%- else %}
      +{% else %}
       websock = ws://127.0.1.2:9001
      -{%- endif %}
      +{% endif %}
       xamari = {{ buildout_directory }}/bin/xamari
      -logfile = ${monitor-directory:public}/enb.xlog
      +logfile = ${monitor-directory:public}/gnb.xlog
       inline =
         #!/bin/sh
         exec ${:xamari} xlog --rotate ${:rotatespec} ${:websock} ${:logfile} ${:logspec}
      @@ -177,6 +183,52 @@ wrapper-path = ${directory:service}/${:_buildout_section_name_}
       hash-files = ${:fluentbit-config}
       {% endif %}
      
      +[amarisoft-stats-template]
      +recipe = slapos.recipe.template:jinja2
      +extensions = jinja2.ext.do
      +log-output = ${directory:var}/log/amarisoft-stats.json.log
      +context =
      +  section directory directory
      +  key slapparameter_dict slap-configuration:configuration
      +  key log_file :log-output
      +  raw stats_period {{ slapparameter_dict.get("gnb_stats_fetch_period", 60) }}
      +  raw testing {{ slapparameter_dict.get("testing", False) }}
      +  raw python_path {{ buildout_directory}}/bin/pythonwitheggs
      +mode = 0775
      +url = {{ ru_amarisoft_stats_template }}
      +output = ${directory:bin}/amarisoft-stats.py
      +
      +[amarisoft-rf-info-template]
      +recipe = slapos.recipe.template:jinja2
      +extensions = jinja2.ext.do
      +log-output = ${directory:var}/log/amarisoft-rf-info.json.log
      +context =
      +  section directory directory
      +  key slapparameter_dict slap-configuration:configuration
      +  key log_file :log-output
      +  raw stats_period {{ slapparameter_dict.get("gnb_stats_fetch_period", 60) }}
      +  raw testing {{ slapparameter_dict.get("testing", False) }}
      +  raw python_path {{ buildout_directory}}/bin/pythonwitheggs
      +mode = 0775
      +url = {{ ru_amarisoft_rf_info_template }}
      +output = ${directory:bin}/amarisoft-rf-info.py
      +
      +[amarisoft-stats-service]
      +recipe = slapos.cookbook:wrapper
      +command-line = ${amarisoft-stats-template:output}
      +wrapper-path = ${directory:service}/amarisoft-stats
      +mode = 0775
      +hash-files =
      +  ${amarisoft-stats-template:output}
      +
      +[amarisoft-rf-info-service]
      +recipe = slapos.cookbook:wrapper
      +command-line = ${amarisoft-rf-info-template:output}
      +wrapper-path = ${directory:service}/amarisoft-rf-info
      +mode = 0775
      +hash-files =
      +  ${amarisoft-rf-info-template:output}
      +
       [config-base]
       recipe = slapos.recipe.template:jinja2
       extensions = jinja2.ext.do
      @@ -190,53 +242,47 @@ context =
         raw gtp_addr_v4 {{ lan_ipv4 }}
         raw tx_gain {{ RF.tx_gain }}
         raw rx_gain {{ RF.rx_gain }}
      -  raw earfcn  {{ RF.dl_earfcn }}
      +  raw nr_arfcn {{ RF.dl_nr_arfcn }}
      +  raw nr_band {{ RF.nr_band }}
         raw software_name {{ software_name }}
         raw rf_mode {{ rf_mode }}
         raw trx {{ trx }}
         raw bbu {{ bbu }}
         raw ru_type {{ ru }}
      -  json do_lte true
      -  json do_nr  false
      +  json do_lte false
      +  json do_nr  true
         import  netaddr netaddr
         ${:extra-context}
      
      -[sib-config]
      -<= config-base
      -url = {{ sib23_template }}
      -output = ${directory:etc}/sib23.cfg
      -
       [drb-config]
       <= config-base
      -url = {{ drb_lte_template }}
      +url = {{ drb_nr_template }}
       output = ${directory:etc}/drb.cfg
      
      -[enb-config]
      +[gnb-config]
       <= config-base
      -{% if slapparameter_dict.get("enb_config_link", None) %}
      -url = ${enb-config-dl:target}
      +{% if slapparameter_dict.get("gnb_config_link", None) %}
      +url = ${gnb-config-dl:target}
       {% else %}
       url = {{ enb_template }}
       {% endif %}
      -output = ${directory:etc}/enb.cfg
      +output = ${directory:etc}/gnb.cfg
       extra-context =
           import json_module json
      -    json cell_list {{ rulib.cell_list | tojson }}
      -    key sib23_file sib-config:output
           key drb_file   drb-config:output
       import-list =
           rawfile slaplte.jinja2 {{ slaplte_template }}
      
      -
       [publish-connection-information]
       <= monitor-publish
       recipe = slapos.cookbook:publish.serialised
       {%- if slapparameter_dict.get("websocket_password", "") %}
       websocket_url = ws://[${slap-configuration:ipv6-random}]:9001
       {%- endif %}
      -enb-ipv6 = ${slap-configuration:ipv6-random}
      -enb-ipv4 = {{ lan_ipv4 }}
      -current-earfcn  = {{ RF.dl_earfcn }}
      +gnb-ipv6 = ${slap-configuration:ipv6-random}
      +gnb-ipv4 = {{ lan_ipv4 }}
      +current-nr-arfcn = {{ RF.dl_nr_arfcn }}
      +current-nr-band = {{ RF.nr_band }}
       amarisoft-version = {{ lte_version }}
       license-expiration = {{ lte_expiration }}
       monitor-gadget-url = ${:monitor-base-url}/gadget/software.cfg.html
      @@ -253,10 +299,35 @@ password = {{ slapparameter_dict['monitor-password'] | string }}
       <= monitor-promise-base
       name = ${:_buildout_section_name_}
      
      +[check-sdr-busy.py]
      +<= macro.promise
      +promise = check_sdr_busy
      +config-testing = {{ slapparameter_dict.get("testing", False) }}
      +config-sdr = {{ sdr }}
      +config-sdr_dev  = 0
      +config-dma_chan = 0
      +
       [check-baseband-latency.py]
       <= macro.promise
       promise = check_baseband_latency
       config-testing = {{ slapparameter_dict.get("testing", False) }}
      -config-amarisoft-stats-log = ${ru_amarisoft-stats-template:log-output}
      -config-stats-period = {{ slapparameter_dict.get("enb_stats_fetch_period", 60) }}
      +config-amarisoft-stats-log = ${amarisoft-stats-template:log-output}
      +config-stats-period = {{ slapparameter_dict.get("gnb_stats_fetch_period", 60) }}
       config-min-rxtx-delay = {{ slapparameter_dict.get("min_rxtx_delay", 0) }}
      +
      +[check-amarisoft-stats-log.py]
      +<= macro.promise
      +promise = check_amarisoft_stats_log
      +output = ${directory:plugins}/check-amarisoft-stats-log.py
      +config-testing = {{ slapparameter_dict.get("testing", False) }}
      +config-amarisoft-stats-log = ${amarisoft-stats-template:log-output}
      +config-stats-period = {{ slapparameter_dict.get("gnb_stats_fetch_period", 60) }}
      +
      +[check-rx-saturated.py]
      +<= macro.promise
      +promise = check_rx_saturated
      +config-testing = {{ slapparameter_dict.get("testing", False) }}
      +config-rf-rx-chan-list = {{ list(range(0, int(slapparameter_dict.get('n_antenna_ul', default_n_antenna_ul)))) }}
      +config-amarisoft-stats-log = ${amarisoft-stats-template:log-output}
      +config-stats-period = {{ slapparameter_dict.get("gnb_stats_fetch_period", 60) }}
      +config-max-rx-sample-db = {{ slapparameter_dict.get("max_rx_sample_db", 0) }}
      ```
      503e50a4
    • Kirill Smelkov's avatar
      software/ors-amarisoft: Start to introduce ORS mode · f3f1cb46
      Kirill Smelkov authored
      In MultiRU we are going to have generic software with flexible configuration
      and many features + ORS software, which wraps the generic part, adds ORS
      specific features, and translates simple ORS schema to generic ones.
      
      Let's first move ORS-specific bits to dedicated place as a preparatory step for that:
      
      - add software-ors.cfg . In the future software-ors.cfg will be the entry point
        for ORS mode, but for now it is only a place from where software-ors-* extend from.
      
      - add instance-ors.cfg . We move ORS-specific code from instance.cfg here and
        set it to use instance-ors-enb.jinja2.cfg when instantiating enb/gnb.
      
      - add instance-ors-enb.jinja2.cfg . This pulls ORS-specific code from
        instance-enb.jinja2.cfg + instance-gnb.jinja2.cfg and wraps them with ORS
        context built by instance-ors.cfg
      
      - in the templates the way to see whether it is ORS or not, and if yes, to access
        ORS parameters is changed: now it is always via checking `ors` object, which
        can be either False for non-ORS mode, or dict for ORS mode, and if it is dict
        the keys inside that dict are ORS specific properties, e.g. ors['one-watt'].
      
        This adjustment is handy to organize ORS mode extension from generic with
        generic setting `json ors false` in the context, and ORS mode undoing that and
        setting ORS context properly. I think the code also reads more clear this way, i.e. with
      
       	{%- if one_watt == "True" %}
      
        changed to
      
      	{%- if ors['one-watt'] %}
      
        because in the second variant it is clear that one-watt is a property of ORS.
      
      The rendered configs for enb.cfg and gnb.cfg stay unchanged.
      
      /cc @jhuge, @lu.xu, @tomo, @xavier_thompson, @Daetalus
      /proposed-for-review-on nexedi/slapos!1522
      /reviewed-by TrustMe
      f3f1cb46
    • Kirill Smelkov's avatar
      software/ors-amarisoft: Rename ors-id -> comp-id · a6a4d2d4
      Kirill Smelkov authored
      We use computer identifier in fluentd forwarding. This identifier was named as
      ors-id, but it applies not only to ORS, but also to generic deployment with
      BBU.
      
      In the next patch we are going to introduce ORS mode and move ORS-specific
      bits to software-ors.cfg and instance-ors.cfg + co. The computer identifier
      will stay in generic part, because it is generic.
      
      -> To avoid ambiguity rename it to be clearly unrelated to ORS specifics.
      
      /cc @jhuge, @lu.xu, @tomo, @xavier_thompson, @Daetalus
      /proposed-for-review-on nexedi/slapos!1522
      /reviewed-by TrustMe
      a6a4d2d4
    • Kirill Smelkov's avatar
      software/ors-amarisoft: enb.jinja2.cfg: Tune PUCCH 1 and PUCCH 3 for CA · eab4a41b
      Kirill Smelkov authored
      When there are multiple cells and CA configured, without those parameters enb fails to start with e.g.
      
          n1_pucch_an_cs_count must be > 0 for the CA Primary cell
      
      Set them on all cell like illustrated in enb-2cc.cfg and enb-3cc.cfg Amarisoft examples.
      
      Not sure selected values are best - I took them from above mentioned examples as is.
      
      Reference to documentation about those parameters:
      
      https://tech-academy.amarisoft.com/lteenb.doc#prop.pucch_dedicated.n1_pucch_an_cs_count
      https://tech-academy.amarisoft.com/lteenb.doc#prop.pucch_dedicated.n3_pucch_an_n_rb
      https://tech-academy.amarisoft.com/lteenb.doc#prop.pucch_dedicated.ack_nack_feedback_mode_ca
      
      /cc @lu.xu, @tomo, @xavier_thompson, @Daetalus
      /helped-by @jhuge
      /reviewed-on nexedi/slapos!1519
      eab4a41b
    • Jérome Perrin's avatar
      stack/erp5: version up erp5diff 0.8.1.9 · 67907147
      Jérome Perrin authored
      67907147
  4. 26 Jan, 2024 3 commits
  5. 25 Jan, 2024 3 commits