Commit 37b1b28d authored by Kirill Smelkov's avatar Kirill Smelkov

software/ors-amarisoft: enb: Take frequencies control into our hands

Currently when setting up cells we allow users to input

- dl_earfcn for LTE, and
- dl_nr_arfcn and nr_band for NR

and from that lteenb automatically computes

- ul_earfcn for LTE, and
- ul_nr_arfcn and ssb_nr_arfcn for NR

everything kind of works out of the box for eNB case when there is simple SDR Radio Unit.

Then there are also the following cases:

1. we also need to set UL frequency when configuring Lopcomm RU, or any other ORAN-based Radio Unit.
2. we also need to specify DL/UL frequencies in MHz when configuring Lopcomm RU.
3. when configuring NR peercell it is required to set ssb_nr_arfcn. It is also
   required to set both dl_nr_arfcn and ul_nr_arfcn, and so far we were setting
   only DL one and using it for both assuming TDD band.
4. when configuring UEsim we need to specify both dl_nr_arfcn and ssb_nr_arfcn for NR cells.

So the problem is that even though lteenb automatically computes UL and SSB
frequencies, there is no way for us to reuse results of that computation for
scenarios outside of "simple enb" case.

As the result we kind of workaround that currently with exposing additional
parameters and asking users to look into enb, probably its `cell phy` output,
with the following:

     ---- 8< ---- (from UEsim)
     "ssb_nr_arfcn": {
       "title": "SSB NR ARFCN",
       "description": "SSB NR ARFCN, you can retrieve from ENB/GNB side",

or use additional parameters just for ul_earfcn and frequency-in-MHz:

     ---- 8< ---- (from ru/lopcomm)
     "txa0cc00_center_frequency": {
       "title": "Lopcomm ORAN DL Center Frequency in MHz (TXA0CC00)",
       "description": "Lopcomm ORAN Center Frequency in MHz (TXA0CC00)",
       "type": "number",
       "default": 2140
     },
     "rxa0cc00_center_frequency_earfcn": {
       "title": "Lopcomm ORAN UL Center Frequency EARFCN (RXA0CC00)",
       "description": "Lopcomm ORAN Center Frequency EARFCN (RXA0CC00)",
       "type": "number",
       "default": 18300
     },
     "rxa0cc00_center_frequency": {
       "title": "Lopcomm ORAN UL Center Frequency in MHz (RXA0CC00)",
       "description": "Lopcomm ORAN Center Frequency in MHz (RXA0CC00)",
       "type": "number",
       "default": 1950
     },

which prevents automation, opens the door for inconsistencies and puts the load
to resolve all that on users.

The root cause of the problem is that there is no way to access at
instantiation time what lteenb would compute internally. But DL<->UL conversion
and DL->SSB conversion is not a difficult task and we can do that on our own.

-> Do that here and solve all the problems listed above in one go.

For frequency computations we use my xlte.earfcn and xlte.nrarfcn modules and
upstream nrarfcn egg. For this xlte is updated(*) to primarily pick up

kirr/xlte@6cb9d37f
kirr/xlte@b8065120

and the rest of the conversion is in slaplte to use corresponding dl2ul and
dl2ssb routines.

For consistency ul_earfcn and ul_nr_arfcn now become input parameters for LTE
and NR cells, but optional. If they are absent - they are computed with
defaults, but a user can now control them explicitly. The same applies for
ssb_nr_arfcn.

This patch does not convert UEsim yet, because UEsim does not use rulib yet and
will be handled all in one go in a follow-up step.

The computation routines are thoroughly tested. First they have unit tests
inside XLTE itself, then we also update our tests in generic test/test.py here
with explicitly checking that correct numbers are emitted for UL and SSB
frequencies, and third I've also verified SSB computation results with respect
to https://tech-academy.amarisoft.com/OutOfBox_UEsim_SA.html#Tips_SSB_Frequency .

All this creates a base to be sure that the computations are correct and we are
indeed safe to switch our frequencies computation modules.

(*) full upgrade brings kirr/xlte@e716ab51...8e606c64 ;
    nrarfcn egg: https://pypi.org/project/nrarfcn/
parent 7a22aaa0
......@@ -54,6 +54,16 @@
"description": "NR band number",
"type": "integer"
},
"ul_nr_arfcn": {
"title": "UL NR ARFCN",
"description": "Uplink NR Absolute Radio Frequency Channel Number of the cell. By default a frequency corresponding to dl_nr_arfcn and nr_band is chosen.",
"type": "integer"
},
"ssb_nr_arfcn": {
"title": "SSB NR ARFCN",
"description": "SSB NR Absolute Radio Frequency Channel Number of the cell. If set it must be an element of global synchronization raster and be at offset from center DL frequency that aligns with SSB subcarrier spacing of selected band. By default a valid frequency nearby dl_nr_arfcn is chosen.",
"type": "integer"
},
"ssb_pos_bitmap": {
"title": "SSB Position Bitmap",
"description": "SSB position bitmap in bits (4, 8 or 64 bits depending on the DL frequency).",
......@@ -68,11 +78,6 @@
},
"$defs": {
"ssb_nr_arfcn": {
"title": "SSB NR ARFCN",
"description": "SSB NR Absolute Radio Frequency Channel Number of the cell",
"type": "number"
},
"tac": {
"title": "Tracking Area Code",
"description": "Integer (range 0 to 16777215)",
......
......@@ -51,7 +51,7 @@
dl_nr_arfcn: {{ ncell.dl_nr_arfcn }},
band: {{ ncell.nr_band }},
ssb_nr_arfcn: {{ ncell.ssb_nr_arfcn }},
ul_nr_arfcn: {{ ncell.dl_nr_arfcn }}, {#- XXX assumes nr_band is TDD #}
ul_nr_arfcn: {{ ncell.ul_nr_arfcn }},
tac: {{ ncell.tac }},
ssb_subcarrier_spacing: 30,
ssb_period: 20,
......@@ -221,6 +221,7 @@
n_id_cell: {{ cell.pci }},
root_sequence_index: {{ cell.root_sequence_index }},
dl_earfcn: {{ cell.dl_earfcn }},
ul_earfcn: {{ cell.ul_earfcn }},
inactivity_timer: {{ cell.inactivity_timer }},
// Handover
......@@ -440,9 +441,11 @@
n_id_cell: {{ cell.pci }},
band: {{ cell.nr_band }},
dl_nr_arfcn: {{ cell.dl_nr_arfcn }},
ul_nr_arfcn: {{ cell.ul_nr_arfcn }},
bandwidth: {{ cell.bandwidth }},
subcarrier_spacing: 30,
ssb_nr_arfcn: {{ cell.ssb_nr_arfcn }},
ssb_pos_bitmap: "{{ cell.ssb_pos_bitmap }}",
root_sequence_index: {{ cell.root_sequence_index }},
......
......@@ -29,6 +29,9 @@ context =
import xbuildout xbuildout
import json_module json
import netaddr netaddr
import nrarfcn_module nrarfcn
import xearfcn_module xlte.earfcn
import xnrarfcn_module xlte.nrarfcn
key eggs_directory buildout:eggs-directory
key develop_eggs_directory buildout:develop-eggs-directory
raw buildout_directory ${buildout:directory}
......@@ -50,6 +53,19 @@ import-list =
[activate-eggs]
recipe = slapos.recipe.build
init =
import pkg_resources as rpkg
buildout = self.buildout['buildout']
env = rpkg.Environment([buildout['develop-eggs-directory'],
buildout['eggs-directory']])
env.scan()
def activate(pkgspec):
req = rpkg.Requirement.parse(pkgspec)
for dist in rpkg.working_set.resolve([req], env):
rpkg.working_set.add(dist)
activate('xlte')
activate('nrarfcn')
# ~ import xbuildout
import sys, types
def readfile(path):
......
......@@ -13,7 +13,6 @@
"nr_cell_id",
"gnb_id_bits",
"dl_nr_arfcn",
"ssb_nr_arfcn",
"nr_band"
],
......@@ -36,7 +35,8 @@
},
"dl_nr_arfcn": { "$ref": "../../../cell/nr/input-schema.json#/properties/dl_nr_arfcn" },
"nr_band": { "$ref": "../../../cell/nr/input-schema.json#/properties/nr_band" },
"ssb_nr_arfcn": { "$ref": "../../../cell/nr/input-schema.json#/$defs/ssb_nr_arfcn" },
"ssb_nr_arfcn": { "$ref": "../../../cell/nr/input-schema.json#/properties/ssb_nr_arfcn" },
"ul_nr_arfcn": { "$ref": "../../../cell/nr/input-schema.json#/properties/ul_nr_arfcn" },
"pci": { "$ref": "../../../cell/nr/input-schema.json#/properties/pci" },
"tac": { "$ref": "../../../cell/nr/input-schema.json#/$defs/tac" }
}
......
......@@ -297,10 +297,14 @@ recipe = slapos.cookbook:publish.serialised
{{ slap_configuration['slap-software-type'] }} = {{ dumps(root) }}
ru = {{ dumps(ru_ref) }}
{%- if cell.cell_type == 'lte' %}
band = {{ dumps('b%d' % xearfcn_module.band(cell.dl_earfcn)[0].band) }}
dl_earfcn = {{ dumps(cell.dl_earfcn) }}
ul_earfcn = {{ dumps(cell.ul_earfcn) }}
{%- elif cell.cell_type == 'nr' %}
band = {{ dumps('n%d' % cell.nr_band) }}
dl_nr_arfcn = {{ dumps(cell.dl_nr_arfcn) }}
ul_nr_arfcn = {{ dumps(cell.ul_nr_arfcn) }}
ssb_nr_arfcn= {{ dumps(cell.ssb_nr_arfcn) }}
{%- else %}
{%- do bug('unreachable') %}
{%- endif %}
......
......@@ -91,8 +91,14 @@
<!-- TODO support multiple cells over 1 RU -->
{%- if cell.cell_type == 'lte' %}
{%- set dl_arfcn = cell.dl_earfcn %}
{%- set ul_arfcn = cell.ul_earfcn %}
{%- set dl_freq = int(xearfcn_module.frequency(dl_arfcn) * 1e6) %}
{%- set ul_freq = int(xearfcn_module.frequency(ul_arfcn) * 1e6) %}
{%- elif cell.cell_type == 'nr' %}
{%- set dl_arfcn = cell.dl_nr_arfcn %}
{%- set ul_arfcn = cell.ul_nr_arfcn %}
{%- set dl_freq = int(xnrarfcn_module.frequency(dl_arfcn) * 1e6) %}
{%- set ul_freq = int(xnrarfcn_module.frequency(ul_arfcn) * 1e6) %}
{%- else %}
{%- do bug('unreachable') %}
{%- endif %}
......@@ -100,7 +106,7 @@
<tx-array-carriers>
<name>{{ TxCarrier }}</name>
<absolute-frequency-center>{{ dl_arfcn }}</absolute-frequency-center>
<center-of-channel-bandwidth>{{ 1000000*ru.get('txa0cc00_center_frequency', 2140) }}</center-of-channel-bandwidth>
<center-of-channel-bandwidth>{{ dl_freq }}</center-of-channel-bandwidth>
<channel-bandwidth>{{ bw }}</channel-bandwidth>
<active>{{ ru.txrx_active }}</active>
<rw-type>{{ cell.cell_type | upper }}</rw-type>
......@@ -111,8 +117,8 @@
</tx-array-carriers>
<rx-array-carriers>
<name>{{ RxCarrier }}</name>
<absolute-frequency-center>{{ ru.get('rxa0cc00_center_frequency_earfcn', 18300) }}</absolute-frequency-center>
<center-of-channel-bandwidth>{{ 1000000*ru.get('rxa0cc00_center_frequency', 1950) }}</center-of-channel-bandwidth>
<absolute-frequency-center>{{ ul_arfcn }}</absolute-frequency-center>
<center-of-channel-bandwidth>{{ ul_freq }}</center-of-channel-bandwidth>
<channel-bandwidth>{{ bw }}</channel-bandwidth>
<active>{{ ru.txrx_active }}</active>
<downlink-radio-frame-offset>0</downlink-radio-frame-offset>
......
......@@ -63,25 +63,6 @@
}
},
"txa0cc00_center_frequency": {
"title": "Lopcomm ORAN DL Center Frequency in MHz (TXA0CC00)",
"description": "Lopcomm ORAN Center Frequency in MHz (TXA0CC00)",
"type": "number",
"default": 2140
},
"rxa0cc00_center_frequency_earfcn": {
"title": "Lopcomm ORAN UL Center Frequency EARFCN (RXA0CC00)",
"description": "Lopcomm ORAN Center Frequency EARFCN (RXA0CC00)",
"type": "number",
"default": 18300
},
"rxa0cc00_center_frequency": {
"title": "Lopcomm ORAN UL Center Frequency in MHz (RXA0CC00)",
"description": "Lopcomm ORAN Center Frequency in MHz (RXA0CC00)",
"type": "number",
"default": 1950
},
"reset_schedule": {
"title": "Cron schedule for RRH reset",
"description": "Refer https://crontab.guru/ to make a reset schedule for RRH, for example, '0 1 * * *' means the RRH will reset every day at 1 am",
......
......@@ -73,6 +73,8 @@ url = {{ ru_lopcomm_cu_config_template }}
{% endif %}
output = ${directory:etc}/{{B('%s-cu_config.xml' % ru_ref)}}
extra-context =
import xearfcn_module xlte.earfcn
import xnrarfcn_module xlte.nrarfcn
key ru :ru
key cell :cell
ru = {{ dumps(ru) }}
......
......@@ -104,6 +104,24 @@
#}
{%- set J = json_module.loads %}
{#- jdefault_ul_earfcn returns default UL EARFCN corresponding to DL EARFCN. #}
{%- macro jdefault_ul_earfcn(dl_earfcn) %}
{{- xearfcn_module.dl2ul(dl_earfcn) | tojson }}
{%- endmacro %}
{#- jdefault_ul_nr_arfcn returns default UL NR ARFCN corresponding to DL NR ARFCN and band. #}
{%- macro jdefault_ul_nr_arfcn(dl_nr_arfcn, nr_band) %}
{{- xnrarfcn_module.dl2ul(dl_nr_arfcn, nr_band) | tojson }}
{%- endmacro %}
{#- jdefault_ssb_nr_arfcn returns default SSB NR ARFCN corresponding to DL NR ARFCN
and band. #}
{%- macro jdefault_ssb_nr_arfcn(dl_nr_arfcn, nr_band) %}
{#- NOTE: computations rechecked wrt https://tech-academy.amarisoft.com/OutOfBox_UEsim_SA.html#Tips_SSB_Frequency #}
{%- set ssb_nr_arfcn, max_ssb_scs_khz = xnrarfcn_module.dl2ssb(dl_nr_arfcn, nr_band) %}
{{- ssb_nr_arfcn | tojson }}
{%- endmacro %}
{#- tap indicates tap interface, that slapos told us to use,
or 'xxx-notap-xxx' if slapos provided us either nothing or empty string. #}
......@@ -284,6 +302,14 @@
{%- endfor %}
{%- set n = len(list(icell_dict|dictsort | selectattr('1._.cell_type', '==', cell.cell_type))) %}
{%- do cell.setdefault('root_sequence_index', 1 + 203*(cell.cell_type == 'lte') + n) %}
{%- if cell.cell_type == 'lte' %}
{%- do cell.setdefault('ul_earfcn', J(jdefault_ul_earfcn(cell.dl_earfcn))) %}
{%- elif cell.cell_type == 'nr' %}
{%- do cell.setdefault('ul_nr_arfcn', J(jdefault_ul_nr_arfcn(cell.dl_nr_arfcn, cell.nr_band))) %}
{%- do cell.setdefault('ssb_nr_arfcn', J(jdefault_ssb_nr_arfcn(cell.dl_nr_arfcn, cell.nr_band))) %}
{%- else %}
{%- do bug('unreachable') %}
{%- endif %}
{%- endmacro %}
......@@ -347,6 +373,14 @@
{%- set _ = ishared['_'] %}
{%- if 'cell_type' in _ and _.get('cell_kind') == 'enb_peer' %}
{%- set ipeercell = ishared %}
{%- if _.cell_type == 'lte' %}
{%- do _.setdefault('ul_earfcn', J(jdefault_ul_earfcn(_.dl_earfcn))) %}
{%- elif _.cell_type == 'nr' %}
{%- do _.setdefault('ul_nr_arfcn', J(jdefault_ul_nr_arfcn(_.dl_nr_arfcn, _.nr_band))) %}
{%- do _.setdefault('ssb_nr_arfcn', J(jdefault_ssb_nr_arfcn(_.dl_nr_arfcn, _.nr_band))) %}
{%- else %}
{%- do bug('unreachable') %}
{%- endif %}
{%- do ipeercell_dict.update({ref: ipeercell}) %}
{%- else %}
{%- do qother.append(ishared) %}
......
......@@ -28,6 +28,9 @@ def j2render(src, out, jcfg):
textctx += 'json %s %s\n' % (k, json.dumps(v))
textctx += 'import xbuildout xbuildout\n'
textctx += 'import json_module json\n'
textctx += 'import nrarfcn_module nrarfcn\n'
textctx += 'import xearfcn_module xlte.earfcn\n'
textctx += 'import xnrarfcn_module xlte.nrarfcn\n'
buildout = None # stub
r = jinja2_template.Recipe(buildout, "recipe", {
'extensions': 'jinja2.ext.do',
......@@ -469,7 +472,6 @@ def _do_enb_with(iru_icell_func):
'nr_cell_id': '0x77712',
'gnb_id_bits': 22,
'dl_nr_arfcn': 520000,
'ssb_nr_arfcn': 520090,
'nr_band': 38,
'pci': 75,
'tac': 321,
......
......@@ -156,6 +156,7 @@ eggs =
xmltodict
ncclient
${lxml-python:egg}
nrarfcn
netifaces
netaddr
interpreter = pythonwitheggs
......@@ -163,7 +164,7 @@ interpreter = pythonwitheggs
[xlte-repository]
recipe = slapos.recipe.build:gitclone
repository = https://lab.nexedi.com/kirr/xlte.git
revision = e716ab51
revision = 8e606c64
git-executable = ${git:location}/bin/git
[xlte]
......@@ -192,3 +193,4 @@ exe = ${dnsmasq:location}/sbin/dnsmasq
websocket-client = 1.4.2
ncclient = 0.6.13
xmltodict = 0.13.0
nrarfcn = 2.4.0:whl
......@@ -274,18 +274,18 @@ class RFTestCase4(AmariTestCase):
assertMatch(t, q('RU3'), {'tx_gain': 13, 'rx_gain': 23, 'txrx_active': 'ACTIVE'})
assertMatch(t, q('RU4'), {'tx_gain': 14, 'rx_gain': 24, 'txrx_active': 'ACTIVE'})
assertMatch(t, q('RU1.CELL'), dict(
dl_earfcn= 100,
dl_nr_arfcn=NO))
assertMatch(t, q('RU2.CELL'), dict(
dl_earfcn=40200,
dl_nr_arfcn=NO))
assertMatch(t, q('RU1.CELL'), dict(band='b1',
dl_earfcn= 100, ul_earfcn=18100,
dl_nr_arfcn=NO, ul_nr_arfcn=NO, ssb_nr_arfcn=NO))
assertMatch(t, q('RU2.CELL'), dict(band='b41',
dl_earfcn=40200, ul_earfcn=40200,
dl_nr_arfcn=NO, ul_nr_arfcn=NO, ssb_nr_arfcn=NO))
assertMatch(t, q('RU3.CELL'), dict(band='n74',
dl_earfcn=NO,
dl_nr_arfcn=300300))
dl_earfcn=NO, ul_earfcn=NO,
dl_nr_arfcn=300300, ul_nr_arfcn=290700, ssb_nr_arfcn=300270))
assertMatch(t, q('RU4.CELL'), dict(band='n40',
dl_earfcn=NO,
dl_nr_arfcn=470400))
dl_earfcn=NO, ul_earfcn=NO,
dl_nr_arfcn=470400, ul_nr_arfcn=470400, ssb_nr_arfcn=470430))
# ENBTestCase4 provides base class for unit-testing eNB service.
......@@ -337,7 +337,7 @@ class ENBTestCase4(RFTestCase4):
_('PEER5', XN_PEER('55.1.1.1'))
_('PEERCELL4', LTE(700) | LTE_PEER(0x12345, 35, 0x123))
_('PEERCELL5', NR(520000,38) | NR_PEER(0x77712,22, 75, 0x321) | {'ssb_nr_arfcn': 520090})
_('PEERCELL5', NR(520000,38) | NR_PEER(0x77712,22, 75, 0x321))
def CELLcfg(i):
return CENB(i, 0x10+i) | TAC(0x100+i) | {
......@@ -370,14 +370,14 @@ class ENBTestCase4(RFTestCase4):
assertMatch(t, t.enb_cfg['cell_list'], [
dict( # CELL1
uldl_config=NO, rf_port=0, n_antenna_dl=4, n_antenna_ul=2,
dl_earfcn=100,
dl_earfcn=100, ul_earfcn=18100,
n_rb_dl=25,
cell_id=0x1, n_id_cell=0x11, tac=0x101,
root_sequence_index=101, inactivity_timer=1001,
),
dict( # CELL2
uldl_config=2, rf_port=1, n_antenna_dl=4, n_antenna_ul=2,
dl_earfcn=40200,
dl_earfcn=40200, ul_earfcn=40200,
n_rb_dl=50,
cell_id=0x2, n_id_cell=0x12, tac=0x102,
root_sequence_index=102, inactivity_timer=1002,
......@@ -387,7 +387,7 @@ class ENBTestCase4(RFTestCase4):
assertMatch(t, t.enb_cfg['nr_cell_list'], [
dict( # CELL3
tdd_ul_dl_config=NO, rf_port=2, n_antenna_dl=4, n_antenna_ul=2,
dl_nr_arfcn=300300, band=74,
dl_nr_arfcn=300300, ul_nr_arfcn=290700, ssb_nr_arfcn=300270, band=74,
bandwidth=15,
cell_id=0x3, n_id_cell=0x13, tac=NO,
root_sequence_index=103, inactivity_timer=1003,
......@@ -398,7 +398,7 @@ class ENBTestCase4(RFTestCase4):
period=5, dl_slots=7, dl_symbols=6, ul_slots=2, ul_symbols=4,
)},
rf_port=3, n_antenna_dl=4, n_antenna_ul=2,
dl_nr_arfcn=470400, band=40,
dl_nr_arfcn=470400, ul_nr_arfcn=470400, ssb_nr_arfcn=470430, band=40,
bandwidth=20,
cell_id=0x4, n_id_cell=0x14, tac=NO,
root_sequence_index=104, inactivity_timer=1004,
......@@ -465,17 +465,20 @@ class Lopcomm4:
# RU configuration in cu_config.xml
def test_ru_cu_config_xml(t):
def uctx(rf_mode, cell_type, dl_arfcn, bw, tx_gain, rx_gain):
def uctx(rf_mode, cell_type, dl_arfcn, ul_arfcn, bw, dl_freq, ul_freq, tx_gain, rx_gain):
return {
'tx-array-carriers': {
'rw-duplex-scheme': rf_mode,
'rw-type': cell_type,
'absolute-frequency-center': '%d' % dl_arfcn,
'center-of-channel-bandwidth': '%d' % dl_freq,
'channel-bandwidth': '%d' % bw,
'gain': '%d' % tx_gain,
'active': 'ACTIVE',
},
'rx-array-carriers': {
'absolute-frequency-center': '%d' % ul_arfcn,
'center-of-channel-bandwidth': '%d' % ul_freq,
'channel-bandwidth': '%d' % bw,
# XXX no rx_gain
'active': 'ACTIVE',
......@@ -484,11 +487,11 @@ class Lopcomm4:
_ = t._test_ru_cu_config_xml
# rf_mode ctype dl_arfcn bw txg rxg
_(1, uctx('FDD', 'LTE', 100, 5000000, 11, 21))
_(2, uctx('TDD', 'LTE', 40200, 10000000, 12, 22))
_(3, uctx('FDD', 'NR', 300300, 15000000, 13, 23))
_(4, uctx('TDD', 'NR', 470400, 20000000, 14, 24))
# rf_mode ctype dl_arfcn ul_arfcn bw dl_freq ul_freq txg rxg
_(1, uctx('FDD', 'LTE', 100, 18100, 5000000, 2120000000, 1930000000, 11, 21))
_(2, uctx('TDD', 'LTE', 40200, 40200, 10000000, 2551000000, 2551000000, 12, 22))
_(3, uctx('FDD', 'NR', 300300, 290700, 15000000, 1501500000, 1453500000, 13, 23))
_(4, uctx('TDD', 'NR', 470400, 470400, 20000000, 2352000000, 2352000000, 14, 24))
def _test_ru_cu_config_xml(t, i, uctx):
cu_xml = t.ipath('etc/%s' % xbuildout.encode('%s-cu_config.xml' % t.ref('RU%d' % i)))
......
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment