An error occurred fetching the project authors.
  1. 11 Nov, 2024 6 commits
    • Kirill Smelkov's avatar
      amari.drb: Teach Sampler to be multicell-aware · b1adc068
      Kirill Smelkov authored
      Since 2a016d48 (Draft support for E-UTRAN IP Throughput KPI) there was a
      hardcoded limitation that x.drb_stats generation works with 1-cell
      configurations only. However we do use multicell eNB configurations and
      on such configurations `xamari xlog x.drb_stats` was failing on eNB
      side with
      
          raise RuntimeError(("ue #%s belongs to %d cells;  "+
              "but only single-cell configurations are supported") % (ue_id, len(ju(['cells']))))
      
      because an UE might be associated with multiple cells of one eNB
      due to e.g. Carrier Aggregation.
      
      Now, after we did preparatory amari.drb refactoring and taught BitSync
      to be multicell-aware, we can finally remove the limitation and correct
      the Sampler to handle UEs associated with multiple cells at the same
      time.
      
      The handling is mostly straightforward: for every frame the Sampler needs
      to estimate time of active transmission and amount of transmitted bytes.
      The transmitted amount is Σcell(tx_bytes) and we estimate transmission
      time on different cells C₁ and C₂ as
      
          tx_time ∈ [max(t₁,t₂), min(t₁+t₂, δt/tti)]
      
      In other words when transmission to/from UE on C₁ and C₂ is going in
      parallel in one frame, the time it takes to transmit current part of UEs
      data
      
          1) cannot be less than time spent on each cell
             (equals to the maximum if transmission on Ci was fully performed
              in the same subframes where transmission on Cj was active)
      
          2) cannot be more than the sum of time spent transmitting on each cell
             (equals to the sum if transmission on C₁ and C₂ was performed in
              distinct subframes of the frame), and
      
          3) cannot be more than the whole frame
             (t₁+t₂ could become more than that if there is some overlap in
              subframes where C₁ and C₂ transmissions were done)
      
      The patch implements this logic when updating data flows of an UE.
      
      Amari.drb now works for multicell configurations.
      
      There are one-cell unit tests for Sampler already added in 2a016d48 and
      now I manually verified that `xamari xlog x.drb_stats` works ok with
      multicell eNB configurations.
      
      Unittests for Sampler in multicell environment are left as TODO.
      b1adc068
    • Kirill Smelkov's avatar
      amari.drb: Rework _BitSync to be multicell-aware · 80d82d4e
      Kirill Smelkov authored
      To be able to handle multicell configurations rework _BitSync to handle
      UEs that might be associated with multiple cells at the same time, e.g.
      in the case of Carrier Aggregation.
      
      For this BitSync interface is reworked correspondingly, then it uses
      tx_bytes total->percell splitter, we added in the previous patch, then
      it uses BitSync1 helpers, which each work on the stream of one
      particular cell, and then the result is combined back into multicell _Utx.
      
      The internals of new BitSync1 are very similar to original old BitSync implementation.
      The new BitSync, that wraps many BitSync1s, is new.
      
      The Sampler still accepts UEs with one cell only.
      In the next patch we will update the Sampler to handle multicell
      configurations as well.
      80d82d4e
    • Kirill Smelkov's avatar
      amari.drb: Add _CTXBytesSplitter to split total tx_bytes into per-cell parts · 9cd06cb9
      Kirill Smelkov authored
      To be able to compute E-UTRAN IP Throughput amari.drb uses Sampler which
      detects and extracts separate transmission samples from 100Hz log of
      ue_get[stats]. The Sampler, in turn uses help from _BitSync for correct
      operations because Amarisoft LTEENB updates counters for dl_total_bytes
      and dl_tx at different times, and _BitSync synchronizes streams of those
      updates in time. _BitSync itself works by correlating amount of
      transmitted data (tx_bytes) and transport blocks (#tx) and shifting some
      amount of #tx to previous frame based on the correlation. See d102ffaa
      (amari.drb: Start of the package) for details.
      
      This works ok for configurations with 1 cell, but does not work out of the box
      for multicell configurations because #tx is per-cell value, while e.g.
      dl_total_bytes is reported by LTEENB only as a value aggregated over all cells.
      That's why original implementation in d102ffaa had an assert that the
      number of cells an UE is associated with is 1.
      
      -> Implement custom filter that splits overall tx_bytes into per-cell
         parts via heuristic based on reported per-cell bitrates to workaround
         that: the more cell bitrate is the more is the part of tx_bytes that
         gets associated to this cell.
      
      In basic implementation the heuristic would be to divide tx_bytes as
      
      	tx_bytes(cell) = tx_bytes·β/Σcells(β)		; β is bitrate of a cell
      
      but given that for every frame _BitSync works by computing things based
      on neighbour frame as well we do it as
      
      	tx_bytes(cell) = tx_bytes·(β₁+β₂)/Σcells(β₁+β₂)
      
      This should make the heuristic a bit more stable.
      
      This patch comes with tx_bytes splitter filter itself only. In the next
      patch we will use _CTXBytesSplitter to implement multicell-awareness for
      _BitSync.
      9cd06cb9
    • Kirill Smelkov's avatar
      amari.drb: Start tracking current bitrate in per-cell UE transmission state · 91967123
      Kirill Smelkov authored
      We will need to know current cell DL/UL bitrate for multicell BitSync in
      the next patches.
      91967123
    • Kirill Smelkov's avatar
      amari.drb: Split _Utx into global and per-cell parts · 26a82c6e
      Kirill Smelkov authored
      Continue preparatory steps to support multicell configurations and for
      that split the class that tracks UE transmission state into global part
      (_Utx) and per-cell parts (_UCtx).
      
      Everywhere else in the code we still assert that the number of cells an UE
      attached to is 1, so no support for multicell yet - only preparatory
      non-functional changes currently.
      26a82c6e
    • Kirill Smelkov's avatar
      amari.drb: Move tracking of #tx and #retx into _Utx · bd57acbb
      Kirill Smelkov authored
      Similarly to previous patch this is preparatory non-functional change to
      support multicell configurations.
      
      Previously the number of transmitted transport blocks was passed around
      as separate argument but with multiple cells each cell will have its own
      information about how many TB were transmitted/received and we will need
      to maintain those .tx and .retx in per-cell data structure.
      
      Start preparing to that by moving .tx and .retx to be tracked in UE
      transmission state instead of being passed around separately.
      
      No support for multicell yet - only preparatory non-functional changes currently.
      bd57acbb
  2. 27 Mar, 2023 1 commit
  3. 08 Mar, 2023 2 commits
    • Kirill Smelkov's avatar
      amari.drb += _IncStats · 78f26e3a
      Kirill Smelkov authored
      An utility class to compute avg/std incrementally.
      
      Thanks to https://www.johndcook.com/blog/standard_deviation/ for the
      recipe of how to do it.
      78f26e3a
    • Kirill Smelkov's avatar
      amari.drb: Start of the package · d102ffaa
      Kirill Smelkov authored
      This package will be used to implement E-UTRAN IP Throughput KPI.
      
      In hereby patch we add `drb.Sampler` that extracts samples of
      transmission bursts from `ue_get[stats]` observations.
      
      Let's go through what E-UTRAN IP Throughput KPI is and how it motivates
      functionality provided by this patch.
      
      Overview of E-UTRAN IP Throughput computation
      ---------------------------------------------
      
      This KPI is defined in TS 32.450 [1] and aggregates transmission volume and
      time over bursts of transmissions from an average UE point of view. It should be
      particularly noted that only the time, during which transmission is going on,
      should be accounted. For example if an UE receives 10KB over 4ms burst and the rest of
      the time there is no transmission to it during, say, 1 minute, the downlink IP
      Throughput for that UE over the minute is 20Mbit/s (= 8·10KB/4ms), not 1.3Kbit/s (= 8·10KB/60s).
      This KPI basically shows what would be the speed to e.g. download a response for
      HTTP request issued from a mobile.
      
      [1] https://www.etsi.org/deliver/etsi_ts/132400_132499/132450/16.00.00_60/ts_132450v160000p.pdf#page=13
      
      To compute IP Throughput we thus need to know Σ of transmitted amount
      of bytes, and Σ of the time of all transmission bursts.
      
      Σ of the bytes is relatively easy to get. eNB already provides close values in
      overall `stats` and in per-UE `ue_get[stats]` messages. However there is no
      anything readily available out-of-the box for Σ of bursts transmission time.
      Thus we need to measure the time of transmission bursts ourselves somehow.
      
      It turns out that with current state of things the only practical way to
      measure it to some degree is to poll eNB frequently with `ue_get[stats]` and
      estimate transmission time based on δ of `ue_get` timestamps.
      
      Let's see how frequently we need to poll to get to reasonably accuracy of resulting throughput.
      
      A common situation for HTTP requests issued via LTE is that response content
      downloading time takes only few milliseconds. For example I used chromium
      network profiler to access various sites via internet tethered from my phone
      and saw that for many requests response content downloading time was e.g. 4ms,
      5ms, 3.2ms, etc. The accuracy of measuring transmission time should be thus in
      the order of millisecond to cover that properly. It makes a real difference for
      reported throughput, if say a download sample with 10KB took 4ms, or it took
      e.g. "something under 100ms". In the first case we know that for that sample
      downlink throughput is 2500KB/s, while in the second case all we know is that
      downlink throughput is "higher than 100KB/s" - a 25 times difference and not
      certain. Similarly if we poll at 10ms rate we would get that throughput is "higher
      than 1000KB/s" - a 2.5 times difference from actual value. The accuracy of 1
      millisecond coincides with TTI time and with how downlink/uplink transmissions
      generally work in LTE.
      
      With the above the scheme to compute IP Throughput looks to be as
      follows: poll eNB at 1000Hz rate for `ue_get[stats]`, process retrieved
      information into per-UE and per-QCI streams, detect bursts on each UE/QCI pair,
      and aggregate `tx_bytes` and `tx_time` from every burst.
      
      It looks to be straightforward, but 1000Hz polling will likely create
      non-negligible additional load on the system and disturb eNB itself
      introducing much jitter and harming its latency requirements. That's probably
      why eNB actually rate-limits WebSocket requests not to go higher than 100Hz -
      the frequency 10 times less compared to what we need to get to reasonable
      accuracy for IP throughput.
      
      Fortunately there is additional information that provides a way to improve
      accuracy of measured `tx_time` even when polled every 10ms at 100Hz rate:
      that additional information is the number of transmitted transport blocks to/from
      an UE. If we know that during 10ms frame it was e.g. 4 transport blocks transmitted
      to the UE, that there were no retransmissions *and* that eNB is not congested, we can
      reasonably estimate that it was actually a 4ms transmission. And if eNB is
      congested we can still say that transmission time is somewhere in `[4ms, 10ms]`
      interval because transmitting each transport block takes 1 TTI. Even if
      imprecise that still provides some information that could be useful.
      
      Also 100Hz polling turns to be acceptable from performance point of view and
      does not disturb the system much. For example on the callbox machine the process,
      that issues polls, takes only about 3% of CPU load and only on one core, and
      the CPU usage of eNB does not practically change and its reported tx/rx latency
      does not change as well. For sure, there is some disturbance, but it appears to
      be small. To have a better idea of what rate of polling is possible, I've made
      an experiment with the poller accessing my own websocket echo server quickly
      implemented in python. Both the poller and the echo server are not optimized,
      but without rate-limiting they could go to 8000Hz frequency with reaching 100%
      CPU usage of one CPU core. That 8000Hz is 80x times more compared to 100Hz
      frequency actually allowed by eNB. This shows what kind of polling
      frequency limit the system can handle, if absolutely needed, and that 100Hz
      turns out to be not so high a frequency. Also the Linux 5.6 kernel, installed
      on the callbox from Fedora32, is configured with `CONFIG_HZ=1000`, which is
      likely helping here.
      
      Implementation overview
      ~~~~~~~~~~~~~~~~~~~~~~~
      
      The scheme to compute E-UTRAN IP Throughput is thus as follows: poll eNB at
      100Hz frequency for `ue_get[stats]` and retrieve information about per-UE/QCI
      streams and the number of transport blocks dl/ul-ed to the UE in question
      during that 10ms frame. Estimate `tx_time` taking into account
      the number of transmitted transport blocks. And estimate whether eNB is congested or
      not based on `dl_use_avg`/`ul_use_avg` taken from `stats`. For the latter we
      also need to poll for `stats` at 100Hz frequency and synchronize
      `ue_get[stats]` and `stats` requests in time so that they both cover the same
      time interval of particular frame.
      
      Then organize the polling process to provide aggregated statistics in the form of
      new `x.drb_stats` message, and teach `xamari xlog` to save that messages to
      `enb.xlog` together with `stats`.  Then further adjust `amari.kpi.LogMeasure`
      and generic `kpi.Measurement` and `kpi.Calc` to handle DRB-related data.
      
      ----------------------------------------
      
      In this patch we provide first building block - `Sampler` that extracts bursts
      of data transmissions from stream of `ue_get[stats]` observations.
      
      Even though main idea behind `Sampler` is relatively straightforward, several
      aspects deserves to be noted:
      
      1. information about transmitted bytes and corresponding transmitted transport
         blocks is emitted by eNB not synchronized in time. The reason here is that,
         for example, for DL a block is transmitted via PDCCH+PDSCH during one TTI, and
         then the base station awaits HARQ ACK/NACK. That ACK/NACK comes later via
         PUCCH or PUSCH. The time window in between original transmission and
         reception of the ACK/NACK is 4 TTIs for FDD and 4-13 TTIs for TDD (*).
         And Amarisoft LTEENB updates counters for dl_total_bytes and dl_tx at
         different times:
      
             ue.erab.dl_total_bytes      - right after sending data on  PDCCH+PDSCH
             ue.cell.{dl_tx,dl_retx}     - after receiving ACK/NACK via PUCCH|PUSCH
      
         this way an update to dl_total_bytes might be seen in one frame (= 10·TTI),
         while corresponding update to dl_tx/dl_retx might be seen in either same, or
         next, or next-next frame.
      
         We bring `δ(tx_bytes)` and `#tx_tb` in sync ourselves via _BitSync.
      
         (*) see e.g. Figure 8.1 in "An introduction to LTE, 2nd ed."
      
      2. when we see multiple transmissions related to UE on different QCIs, we
         cannot directly use corresponding number of transport blocks to estimate
         transmissions times because we do not know how eNB scheduler placed those
         transmissions onto resource map. So without additional information we can only
         estimate corresponding lower and upper bounds.
      d102ffaa