Skip to content

G
gitlab-ce
Commit b3c5c996 authored by Sean McGivern's avatar Sean McGivern
Browse files
Options

Simplify adding new Redis instance 

The main sticking point was `config/README.md`, which went into a lot of
detail about each instance. It was quite easy to make a copy and paste
error, as some instances had default URLs and others didn't, some had
fallback instances and others didn't, etc.

This changes a number of things:

1. `config/README.md` now has a table for the Redis instances with a
   single list of instructions that apply to all instances.
2. There are now specs to ensure that each Redis instance has a default
   URL set when no Redis config at all is present.
3. Instances that use a fallback instance for config will automatically
   use the default URL of their fallback instance. They should not
   define their own default URL.
4. The `.gitignore` file does not need to be updated for each new Redis
   instance.
5. The example config files are now gone as `request.yml.example` should
   be sufficient.
parent 2f89dcdb
Hide whitespace changes
Inline Side-by-side
Showing with 75 additions and 411 deletions
.gitignore
View file @ b3c5c996
...@@ -40,11 +40,7 @@ eslint-report.html ...@@ -40,11 +40,7 @@ eslint-report.html
/config/initializers/smtp_settings.rb /config/initializers/smtp_settings.rb
/config/initializers/relative_url.rb /config/initializers/relative_url.rb
/config/resque.yml /config/resque.yml
/config/redis.cache.yml /config/redis.*.yml
/config/redis.queues.yml
/config/redis.shared_state.yml
/config/redis.trace_chunks.yml
/config/redis.rate_limiting.yml
/config/unicorn.rb /config/unicorn.rb
/config/puma.rb /config/puma.rb
/config/secrets.yml /config/secrets.yml
......
config/README.md
View file @ b3c5c996
# Configuration files Documentation # Configuration files Documentation
Note that most configuration files (`config/*.*`) committed into Note that most configuration files (`config/*.*`) committed into
[gitlab-ce](https://gitlab.com/gitlab-org/gitlab-foss) **will not be used** for [gitlab-foss](https://gitlab.com/gitlab-org/gitlab-foss) **will not be used** for
[omnibus-gitlab](https://gitlab.com/gitlab-org/omnibus-gitlab). Configuration [omnibus-gitlab](https://gitlab.com/gitlab-org/omnibus-gitlab). Configuration
files committed into gitlab-ce are only used for development. files committed into gitlab-foss are only used for development.
## gitlab.yml ## gitlab.yml
You can find most of GitLab configuration settings here. You can find most of the GitLab configuration settings here.
## mail_room.yml ## mail_room.yml
...@@ -21,7 +21,7 @@ This file is called `resque.yml` for historical reasons. We are **NOT** ...@@ -21,7 +21,7 @@ This file is called `resque.yml` for historical reasons. We are **NOT**
using Resque at the moment. It is used to specify Redis configuration using Resque at the moment. It is used to specify Redis configuration
values when a single database instance of Redis is desired. values when a single database instance of Redis is desired.
# Advanced Redis configuration files ## Advanced Redis configuration files
In more advanced configurations of Redis key-value storage, it is desirable In more advanced configurations of Redis key-value storage, it is desirable
to separate the keys by lifecycle and intended use to ease provisioning and to separate the keys by lifecycle and intended use to ease provisioning and
...@@ -40,7 +40,7 @@ If desired, the routing URL provided by these settings can be used with: ...@@ -40,7 +40,7 @@ If desired, the routing URL provided by these settings can be used with:
2. TCP port number for each Redis instance desired 2. TCP port number for each Redis instance desired
3. `database number` for each Redis instance desired 3. `database number` for each Redis instance desired
## Example URL attribute formats for GitLab Redis `.yml` configuration files ### Example URL attribute formats for GitLab Redis `.yml` configuration files
* Unix Socket, default Redis database (0) * Unix Socket, default Redis database (0)
* `url: unix:/path/to/redis.sock` * `url: unix:/path/to/redis.sock`
* `url: unix:/path/to/redis.sock?db=` * `url: unix:/path/to/redis.sock?db=`
...@@ -52,170 +52,37 @@ If desired, the routing URL provided by these settings can be used with: ...@@ -52,170 +52,37 @@ If desired, the routing URL provided by these settings can be used with:
* TCP Socket for Redis on remote host `myserver`, port 6379, database 33 * TCP Socket for Redis on remote host `myserver`, port 6379, database 33
* `url: redis://:mynewpassword@myserver:6379/33` * `url: redis://:mynewpassword@myserver:6379/33`
## redis.cache.yml ## Available configuration files
If configured, `redis.cache.yml` overrides the The Redis instances that can be configured are described in the table below. The
`resque.yml` settings to configure the Redis database instance order of precedence for configuration is described below, where `$NAME` and
used for `Rails.cache` and other volatile non-persistent data which enhances `$FALLBACK_NAME` are the upper-cased instance names from the table, and `$name`
the performance of GitLab. and `$fallback_name` are the lower-cased versions:
Settings here can be overridden by the environment variable
`GITLAB_REDIS_CACHE_CONFIG_FILE` which provides 1. The configuration file pointed to by the `GITLAB_REDIS_$NAME_CONFIG_FILE`
an alternate location for configuration settings. environment variable.
1. The configuration file `redis.$name.yml`.
The order of precedence for the URL used to connect to the Redis instance 1. **If a fallback instance is available**, the configuration file
used for `cache` is: `redis.$fallback_name.yml`.
1. URL from a configuration file pointed to by the 1. The configuration file pointed to by the `GITLAB_REDIS_CONFIG_FILE`
`GITLAB_REDIS_CACHE_CONFIG_FILE` environment variable environment variable.
2. URL from `redis.cache.yml` 1. The configuration file `resque.yml`.
3. URL from a configuration file pointed to by the
`GITLAB_REDIS_CONFIG_FILE` environment variable An example configuration file for Redis is in this directory under the name
4. URL from `resque.yml` `resque.yml.example`.
5. `redis://localhost:6380`
| Name | Fallback instance | Purpose |
The order of precedence for all other configuration settings for `cache` | --- | --- | --- |
are selected from only the first of the following files found (if a setting | `cache` | | Volatile non-persistent data |
is not provided in an earlier file, the remainder of the files are not | `queues` | | Background job processing queues |
searched): | `shared_state` | | Persistent application state |
1. the configuration file pointed to by the | `trace_chunks` | `shared_state` | [CI trace chunks](https://docs.gitlab.com/ee/administration/job_logs.html#incremental-logging-architecture) |
`GITLAB_REDIS_CACHE_CONFIG_FILE` environment variable | `rate_limiting` | `cache` | [Rate limiting](https://docs.gitlab.com/ee/user/admin_area/settings/user_and_ip_rate_limits.html) state |
2. the configuration file `redis.cache.yml`
3. the configuration file pointed to by the If no configuration is found, or no URL is found in the configuration
`GITLAB_REDIS_CONFIG_FILE` environment variable file, the default URL used is:
4. the configuration file `resque.yml`
1. `redis://localhost:6380` for `cache`.
## redis.queues.yml 1. `redis://localhost:6381` for `queues`.
1. `redis://localhost:6382` for `shared_state`.
If configured, `redis.queues.yml` overrides the 1. The URL from the fallback instance for all other instances.
`resque.yml` settings to configure the Redis database instance
used for clients of `::Gitlab::Redis::Queues`.
These queues are intended to be the foundation
of reliable inter-process communication between modules, whether on the same
host node, or within a cluster. The primary clients of the queues are
SideKiq, Mailroom, CI Runner, Workhorse, and push services. Settings here can
be overridden by the environment variable
`GITLAB_REDIS_QUEUES_CONFIG_FILE` which provides an alternate location for
configuration settings.
The order of precedence for the URL used to connect to the Redis instance
used for `queues` is:
1. URL from a configuration file pointed to by the
`GITLAB_REDIS_QUEUES_CONFIG_FILE` environment variable
2. URL from `redis.queues.yml`
3. URL from a configuration file pointed to by the
`GITLAB_REDIS_CONFIG_FILE` environment variable
4. URL from `resque.yml`
5. `redis://localhost:6381`
The order of precedence for all other configuration settings for `queues`
are selected from only the first of the following files found (if a setting
is not provided in an earlier file, the remainder of the files are not
searched):
1. the configuration file pointed to by the
`GITLAB_REDIS_QUEUES_CONFIG_FILE` environment variable
2. the configuration file `redis.queues.yml`
3. the configuration file pointed to by the
`GITLAB_REDIS_CONFIG_FILE` environment variable
4. the configuration file `resque.yml`
## redis.shared_state.yml
If configured, `redis.shared_state.yml` overrides the
`resque.yml` settings to configure the Redis database instance
used for clients of `::Gitlab::Redis::SharedState` such as session state,
and rate limiting.
Settings here can be overridden by the environment variable
`GITLAB_REDIS_SHARED_STATE_CONFIG_FILE` which provides
an alternate location for configuration settings.
The order of precedence for the URL used to connect to the Redis instance
used for `shared_state` is:
1. URL from a configuration file pointed to by the
`GITLAB_REDIS_SHARED_STATE_CONFIG_FILE` environment variable
2. URL from `redis.shared_state.yml`
3. URL from a configuration file pointed to by the
`GITLAB_REDIS_CONFIG_FILE` environment variable
4. URL from `resque.yml`
5. `redis://localhost:6382`
The order of precedence for all other configuration settings for `shared_state`
are selected from only the first of the following files found (if a setting
is not provided in an earlier file, the remainder of the files are not
searched):
1. the configuration file pointed to by the
`GITLAB_REDIS_SHARED_STATE_CONFIG_FILE` environment variable
2. the configuration file `redis.shared_state.yml`
3. the configuration file pointed to by the
`GITLAB_REDIS_CONFIG_FILE` environment variable
4. the configuration file `resque.yml`
## redis.trace_chunks.yml
If configured, `redis.trace_chunks.yml` overrides the
`resque.yml` settings to configure the Redis database instance
used for clients of `::Gitlab::Redis::TraceChunks` which stores CI trace chunks.
Settings here can be overridden by the environment variable
`GITLAB_REDIS_TRACE_CHUNKS_CONFIG_FILE` which provides
an alternate location for configuration settings.
The order of precedence for the URL used to connect to the Redis instance
used for `trace_chunks` is:
1. URL from a configuration file pointed to by the
`GITLAB_REDIS_TRACE_CHUNKS_CONFIG_FILE` environment variable
2. URL from `redis.trace_chunks.yml`
3. URL from a configuration file pointed to by the
`GITLAB_REDIS_CONFIG_FILE` environment variable
4. URL from `resque.yml`
5. `redis://localhost:6383`
The order of precedence for all other configuration settings for `trace_chunks`
are selected from only the first of the following files found (if a setting
is not provided in an earlier file, the remainder of the files are not
searched):
1. the configuration file pointed to by the
`GITLAB_REDIS_TRACE_CHUNKS_CONFIG_FILE` environment variable
2. the configuration file `redis.trace_chunks.yml`
3. the configuration file pointed to by the
`GITLAB_REDIS_SHARED_STATE_CONFIG_FILE` environment variable
4. the configuration file `redis.shared_state.yml`
5. the configuration file pointed to by the
`GITLAB_REDIS_CONFIG_FILE` environment variable
6. the configuration file `resque.yml`
## redis.rate_limiting.yml
If configured, `redis.rate_limiting.yml` overrides the `resque.yml` settings to
configure the Redis database instance used for clients of
`::Gitlab::Redis::RateLimiting` which stores
[rate limiting](https://docs.gitlab.com/ee/user/admin_area/settings/user_and_ip_rate_limits.html)
state.
Settings here can be overridden by the environment variable
`GITLAB_REDIS_RATE_LIMITING_CONFIG_FILE` which provides
an alternate location for configuration settings.
The order of precedence for the URL used to connect to the Redis instance
used for `rate_limiting` is:
1. URL from a configuration file pointed to by the
`GITLAB_REDIS_RATE_LIMITING_CONFIG_FILE` environment variable
2. URL from `redis.rate_limiting.yml`
3. URL from a configuration file pointed to by the
`GITLAB_REDIS_CACHE_CONFIG_FILE` environment variable
4. URL from a configuration file pointed to by the
`GITLAB_REDIS_CONFIG_FILE` environment variable
5. URL from `resque.yml`
6. `redis://localhost:6384`
The order of precedence for all other configuration settings for `rate_limiting`
are selected from only the first of the following files found (if a setting
is not provided in an earlier file, the remainder of the files are not
searched):
1. the configuration file pointed to by the
`GITLAB_REDIS_RATE_LIMITING_CONFIG_FILE` environment variable
2. the configuration file `redis.rate_limiting.yml`
3. the configuration file pointed to by the
`GITLAB_REDIS_CACHE_CONFIG_FILE` environment variable
4. the configuration file `redis.cache.yml`
5. the configuration file pointed to by the
`GITLAB_REDIS_CONFIG_FILE` environment variable
6. the configuration file `resque.yml`
config/redis.cache.yml.example deleted 100644 → 0
View file @ 2f89dcdb
# If you change this file in a merge request, please also create
# a merge request on https://gitlab.com/gitlab-org/omnibus-gitlab/merge_requests
#
development:
url: redis://localhost:6379/10
#
# url: redis://localhost:6380
# sentinels:
# -
# host: localhost
# port: 26380 # point to sentinel, not to redis port
# -
# host: replica2
# port: 26380 # point to sentinel, not to redis port
test:
url: redis://localhost:6379/10
#
# url: redis://localhost:6380
production:
# Redis (single instance)
url: unix:/var/run/redis/redis.cache.sock
##
# Redis + Sentinel (for HA)
#
# Please read instructions carefully before using it as you may lose data:
# http://redis.io/topics/sentinel
#
# You must specify a list of a few sentinels that will handle client connection
# please read here for more information: https://docs.gitlab.com/ee/administration/redis/index.html
##
# url: redis://master:6380
# sentinels:
# -
# host: replica1
# port: 26380 # point to sentinel, not to redis port
# -
# host: replica2
# port: 26380 # point to sentinel, not to redis port
config/redis.queues.yml.example deleted 100644 → 0
View file @ 2f89dcdb
# If you change this file in a merge request, please also create
# a merge request on https://gitlab.com/gitlab-org/omnibus-gitlab/merge_requests
#
development:
url: redis://localhost:6379/11
#
# url: redis://localhost:6381
# sentinels:
# -
# host: localhost
# port: 26381 # point to sentinel, not to redis port
# -
# host: replica2
# port: 26381 # point to sentinel, not to redis port
test:
url: redis://localhost:6379/11
#
# url: redis://localhost:6381
production:
# Redis (single instance)
url: unix:/var/run/redis/redis.queues.sock
##
# Redis + Sentinel (for HA)
#
# Please read instructions carefully before using it as you may lose data:
# http://redis.io/topics/sentinel
#
# You must specify a list of a few sentinels that will handle client connection
# please read here for more information: https://docs.gitlab.com/ee/administration/redis/index.html
##
# url: redis://master:6381
# sentinels:
# -
# host: replica1
# port: 26381 # point to sentinel, not to redis port
# -
# host: replica2
# port: 26381 # point to sentinel, not to redis port
config/redis.rate_limiting.yml.example deleted 100644 → 0
View file @ 2f89dcdb
# If you change this file in a merge request, please also create
# a merge request on https://gitlab.com/gitlab-org/omnibus-gitlab/merge_requests
#
development:
url: redis://localhost:6379/4
#
# url: redis://localhost:6382
# sentinels:
# -
# host: localhost
# port: 26382 # point to sentinel, not to redis port
# -
# host: replica2
# port: 26382 # point to sentinel, not to redis port
test:
url: redis://localhost:6379/14
#
# url: redis://localhost:6382
production:
# Redis (single instance)
url: unix:/var/run/redis/redis.rate_limiting.sock
##
# Redis + Sentinel (for HA)
#
# Please read instructions carefully before using it as you may lose data:
# http://redis.io/topics/sentinel
#
# You must specify a list of a few sentinels that will handle client connection
# please read here for more information: https://docs.gitlab.com/ee/administration/redis/index.html
##
# url: redis://master:6382
# sentinels:
# -
# host: replica1
# port: 26382 # point to sentinel, not to redis port
# -
# host: replica2
# port: 26382 # point to sentinel, not to redis port
config/redis.shared_state.yml.example deleted 100644 → 0
View file @ 2f89dcdb
# If you change this file in a merge request, please also create
# a merge request on https://gitlab.com/gitlab-org/omnibus-gitlab/merge_requests
#
development:
url: redis://localhost:6379/12
#
# url: redis://localhost:6382
# sentinels:
# -
# host: localhost
# port: 26382 # point to sentinel, not to redis port
# -
# host: replica2
# port: 26382 # point to sentinel, not to redis port
test:
url: redis://localhost:6379/12
#
# url: redis://localhost:6382
production:
# Redis (single instance)
url: unix:/var/run/redis/redis.shared_state.sock
##
# Redis + Sentinel (for HA)
#
# Please read instructions carefully before using it as you may lose data:
# http://redis.io/topics/sentinel
#
# You must specify a list of a few sentinels that will handle client connection
# please read here for more information: https://docs.gitlab.com/ee/administration/redis/index.html
##
# url: redis://master:6382
# sentinels:
# -
# host: replica1
# port: 26382 # point to sentinel, not to redis port
# -
# host: replica2
# port: 26382 # point to sentinel, not to redis port
config/redis.trace_chunks.yml.example deleted 100644 → 0
View file @ 2f89dcdb
# If you change this file in a merge request, please also create
# a merge request on https://gitlab.com/gitlab-org/omnibus-gitlab/merge_requests
#
development:
url: redis://localhost:6379/13
#
# url: redis://localhost:6382
# sentinels:
# -
# host: localhost
# port: 26382 # point to sentinel, not to redis port
# -
# host: replica2
# port: 26382 # point to sentinel, not to redis port
test:
url: redis://localhost:6379/13
#
# url: redis://localhost:6382
production:
# Redis (single instance)
url: unix:/var/run/redis/redis.trace_chunks.sock
##
# Redis + Sentinel (for HA)
#
# Please read instructions carefully before using it as you may lose data:
# http://redis.io/topics/sentinel
#
# You must specify a list of a few sentinels that will handle client connection
# please read here for more information: https://docs.gitlab.com/ee/administration/redis/index.html
##
# url: redis://master:6382
# sentinels:
# -
# host: replica1
# port: 26382 # point to sentinel, not to redis port
# -
# host: replica2
# port: 26382 # point to sentinel, not to redis port
lib/gitlab/redis/cache.rb
View file @ b3c5c996
...@@ -4,14 +4,6 @@ module Gitlab ...@@ -4,14 +4,6 @@ module Gitlab
module Redis module Redis
class Cache < ::Gitlab::Redis::Wrapper class Cache < ::Gitlab::Redis::Wrapper
CACHE_NAMESPACE = 'cache:gitlab' CACHE_NAMESPACE = 'cache:gitlab'
private
def raw_config_hash
config = super
config[:url] = 'redis://localhost:6380' if config[:url].blank?
config
end
end end
end end
end end
lib/gitlab/redis/queues.rb
View file @ b3c5c996
...@@ -2,21 +2,12 @@ ...@@ -2,21 +2,12 @@
# We need this require for MailRoom # We need this require for MailRoom
require_relative 'wrapper' unless defined?(::Gitlab::Redis::Wrapper) require_relative 'wrapper' unless defined?(::Gitlab::Redis::Wrapper)
require 'active_support/core_ext/object/blank'
module Gitlab module Gitlab
module Redis module Redis
class Queues < ::Gitlab::Redis::Wrapper class Queues < ::Gitlab::Redis::Wrapper
SIDEKIQ_NAMESPACE = 'resque:gitlab' SIDEKIQ_NAMESPACE = 'resque:gitlab'
MAILROOM_NAMESPACE = 'mail_room:gitlab' MAILROOM_NAMESPACE = 'mail_room:gitlab'
private
def raw_config_hash
config = super
config[:url] = 'redis://localhost:6381' if config[:url].blank?
config
end
end end
end end
end end
lib/gitlab/redis/shared_state.rb
View file @ b3c5c996
...@@ -7,14 +7,6 @@ module Gitlab ...@@ -7,14 +7,6 @@ module Gitlab
USER_SESSIONS_NAMESPACE = 'session:user:gitlab' USER_SESSIONS_NAMESPACE = 'session:user:gitlab'
USER_SESSIONS_LOOKUP_NAMESPACE = 'session:lookup:user:gitlab' USER_SESSIONS_LOOKUP_NAMESPACE = 'session:lookup:user:gitlab'
IP_SESSIONS_LOOKUP_NAMESPACE = 'session:lookup:ip:gitlab2' IP_SESSIONS_LOOKUP_NAMESPACE = 'session:lookup:ip:gitlab2'
private
def raw_config_hash
config = super
config[:url] = 'redis://localhost:6382' if config[:url].blank?
config
end
end end
end end
end end
lib/gitlab/redis/wrapper.rb
View file @ b3c5c996
...@@ -6,6 +6,7 @@ ...@@ -6,6 +6,7 @@
# Rails. # Rails.
require 'active_support/core_ext/hash/keys' require 'active_support/core_ext/hash/keys'
require 'active_support/core_ext/module/delegation' require 'active_support/core_ext/module/delegation'
require 'active_support/core_ext/object/blank'
require 'active_support/core_ext/string/inflections' require 'active_support/core_ext/string/inflections'
# Explicitly load Redis::Store::Factory so we can read Redis configuration in # Explicitly load Redis::Store::Factory so we can read Redis configuration in
...@@ -150,11 +151,35 @@ module Gitlab ...@@ -150,11 +151,35 @@ module Gitlab
def raw_config_hash def raw_config_hash
config_data = fetch_config config_data = fetch_config
if config_data config_hash =
config_data.is_a?(String) ? { url: config_data } : config_data.deep_symbolize_keys if config_data
else config_data.is_a?(String) ? { url: config_data } : config_data.deep_symbolize_keys
{ url: '' } else
{ url: '' }
end
if config_hash[:url].blank?
config_hash[:url] = legacy_fallback_urls[self.class.store_name] || legacy_fallback_urls[self.class.config_fallback.store_name]
end end
config_hash
end
# These URLs were defined for cache, queues, and shared_state in
# code. They are used only when no config file exists at all for a
# given instance. The configuration does not seem particularly
# useful - it uses different ports on localhost - but we cannot
# confidently delete it as we don't know if any instances rely on
# this.
#
# DO NOT ADD new instances here. All new instances should define a
# `.config_fallback`, which will then be used to look up this URL.
def legacy_fallback_urls
{
'Cache' => 'redis://localhost:6380',
'Queues' => 'redis://localhost:6381',
'SharedState' => 'redis://localhost:6382'
}
end end
def fetch_config def fetch_config
......
scripts/prepare_build.sh
View file @ b3c5c996
...@@ -38,21 +38,6 @@ sed -i 's|url:.*$|url: redis://redis:6379|g' config/cable.yml ...@@ -38,21 +38,6 @@ sed -i 's|url:.*$|url: redis://redis:6379|g' config/cable.yml
cp config/resque.yml.example config/resque.yml cp config/resque.yml.example config/resque.yml
sed -i 's|url:.*$|url: redis://redis:6379|g' config/resque.yml sed -i 's|url:.*$|url: redis://redis:6379|g' config/resque.yml
cp config/redis.cache.yml.example config/redis.cache.yml
sed -i 's|url:.*$|url: redis://redis:6379/10|g' config/redis.cache.yml
cp config/redis.queues.yml.example config/redis.queues.yml
sed -i 's|url:.*$|url: redis://redis:6379/11|g' config/redis.queues.yml
cp config/redis.shared_state.yml.example config/redis.shared_state.yml
sed -i 's|url:.*$|url: redis://redis:6379/12|g' config/redis.shared_state.yml
cp config/redis.trace_chunks.yml.example config/redis.trace_chunks.yml
sed -i 's|url:.*$|url: redis://redis:6379/13|g' config/redis.trace_chunks.yml
cp config/redis.rate_limiting.yml.example config/redis.rate_limiting.yml
sed -i 's|url:.*$|url: redis://redis:6379/14|g' config/redis.rate_limiting.yml
if [ "$SETUP_DB" != "false" ]; then if [ "$SETUP_DB" != "false" ]; then
setup_db setup_db
elif getent hosts postgres; then elif getent hosts postgres; then
......
spec/support/redis/redis_shared_examples.rb
View file @ b3c5c996
...@@ -327,6 +327,12 @@ RSpec.shared_examples "redis_shared_examples" do ...@@ -327,6 +327,12 @@ RSpec.shared_examples "redis_shared_examples" do
expect(subject.send(:fetch_config)).to eq false expect(subject.send(:fetch_config)).to eq false
end end
it 'has a value for the legacy default URL' do
allow(subject).to receive(:fetch_config) { false }
expect(subject.send(:raw_config_hash)).to include(url: a_string_matching(%r{\Aredis://localhost:638[012]\Z}))
end
end end
def clear_raw_config def clear_raw_config
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment
GitLab Nexedi Edition | About GitLab | About Nexedi | 沪ICP备2021021310号-2 | 沪ICP备2021021310号-7