Mattermost is an open source, private cloud, Slack-alternative from https://mattermost.com.
Mattermost is an open source, private cloud, Slack-alternative from <https://mattermost.com>.
#### MinIO
#### MinIO
...
@@ -321,7 +321,7 @@ MinIO is an object storage server released under Apache License v2.0. It is comp
...
@@ -321,7 +321,7 @@ MinIO is an object storage server released under Apache License v2.0. It is comp
- Layer: Core Service (Processor)
- Layer: Core Service (Processor)
- Process: `nginx`
- Process: `nginx`
Nginx as an ingress port for all HTTP requests and routes them to the approriate sub-systems within GitLab. We are bundling an unmodified version of the popular open source webserver.
Nginx as an ingress port for all HTTP requests and routes them to the appropriate sub-systems within GitLab. We are bundling an unmodified version of the popular open source webserver.
@@ -6,6 +6,7 @@ easy to maintain, and performant for the end-user.
...
@@ -6,6 +6,7 @@ easy to maintain, and performant for the end-user.
## Rules
## Rules
### Utility Classes
### Utility Classes
As part of the effort for [cleaning up our CSS and moving our components into GitLab-UI](https://gitlab.com/groups/gitlab-org/-/epics/950)
As part of the effort for [cleaning up our CSS and moving our components into GitLab-UI](https://gitlab.com/groups/gitlab-org/-/epics/950)
led by the [GitLab UI WG](https://gitlab.com/gitlab-com/www-gitlab-com/merge_requests/20623) we prefer the use of utility classes over adding new CSS. However, complex CSS can be addressed by adding component classes.
led by the [GitLab UI WG](https://gitlab.com/gitlab-com/www-gitlab-com/merge_requests/20623) we prefer the use of utility classes over adding new CSS. However, complex CSS can be addressed by adding component classes.
...
@@ -31,12 +32,12 @@ New utility classes should be added to [`utilities.scss`](https://gitlab.com/git
...
@@ -31,12 +32,12 @@ New utility classes should be added to [`utilities.scss`](https://gitlab.com/git
#### When should I create component classes?
#### When should I create component classes?
We recommend a "utility-first" approach.
We recommend a "utility-first" approach.
1. Start with utility classes.
1. Start with utility classes.
2. If composing utility classes into a component class removes code duplication and encapsulates a clear responsibility, do it.
2. If composing utility classes into a component class removes code duplication and encapsulates a clear responsibility, do it.
This encourages an organic growth of component classes and prevents the creation of one-off unreusable classes. Also, the kind of classes that emerge from "utility-first" tend to be design-centered (e.g. `.button`, `.alert`, `.card`) rather than domain-centered (e.g. `.security-report-widget`, `.commit-header-icon`).
This encourages an organic growth of component classes and prevents the creation of one-off unreusable classes. Also, the kind of classes that emerge from "utility-first" tend to be design-centered (e.g. `.button`, `.alert`, `.card`) rather than domain-centered (e.g. `.security-report-widget`, `.commit-header-icon`).
Examples of component classes that were created using "utility-first" include:
Examples of component classes that were created using "utility-first" include:
...
@@ -45,8 +46,8 @@ Examples of component classes that were created using "utility-first" include:
...
@@ -45,8 +46,8 @@ Examples of component classes that were created using "utility-first" include:
@@ -49,7 +49,9 @@ provided as a prop to the main component.
...
@@ -49,7 +49,9 @@ provided as a prop to the main component.
Be sure to read about [page-specific JavaScript][page_specific_javascript].
Be sure to read about [page-specific JavaScript][page_specific_javascript].
### Bootstrapping Gotchas
### Bootstrapping Gotchas
#### Providing data from HAML to JavaScript
#### Providing data from HAML to JavaScript
While mounting a Vue application may be a need to provide data from Rails to JavaScript.
While mounting a Vue application may be a need to provide data from Rails to JavaScript.
To do that, provide the data through `data` attributes in the HTML element and query them while mounting the application.
To do that, provide the data through `data` attributes in the HTML element and query them while mounting the application.
...
@@ -83,7 +85,8 @@ document.addEventListener('DOMContentLoaded', () => new Vue({
...
@@ -83,7 +85,8 @@ document.addEventListener('DOMContentLoaded', () => new Vue({
```
```
#### Accessing the `gl` object
#### Accessing the `gl` object
When we need to query the `gl` object for data that won't change during the application's life cyle, we should do it in the same place where we query the DOM.
When we need to query the `gl` object for data that won't change during the application's life cycle, we should do it in the same place where we query the DOM.
By following this practice, we can avoid the need to mock the `gl` object, which will make tests easier.
By following this practice, we can avoid the need to mock the `gl` object, which will make tests easier.
It should be done while initializing our Vue instance, and the data should be provided as `props` to the main component:
It should be done while initializing our Vue instance, and the data should be provided as `props` to the main component:
...
@@ -118,6 +121,7 @@ You can read more about components in Vue.js site, [Component System][component-
...
@@ -118,6 +121,7 @@ You can read more about components in Vue.js site, [Component System][component-
### A folder for the Store
### A folder for the Store
#### Vuex
#### Vuex
Check this [page](vuex.md) for more details.
Check this [page](vuex.md) for more details.
### Mixing Vue and jQuery
### Mixing Vue and jQuery
...
@@ -212,6 +216,7 @@ describe('Todos App', () => {
...
@@ -212,6 +216,7 @@ describe('Todos App', () => {
```
```
### `mountComponent` helper
### `mountComponent` helper
There is a helper in `spec/javascripts/helpers/vue_mount_component_helper.js` that allows you to mount a component with the given props:
There is a helper in `spec/javascripts/helpers/vue_mount_component_helper.js` that allows you to mount a component with the given props:
```javascript
```javascript
...
@@ -225,10 +230,12 @@ const vm = mountComponent(Component, data);
...
@@ -225,10 +230,12 @@ const vm = mountComponent(Component, data);
```
```
### Test the component's output
### Test the component's output
The main return value of a Vue component is the rendered output. In order to test the component we
The main return value of a Vue component is the rendered output. In order to test the component we
need to test the rendered output. [Vue][vue-test] guide's to unit test show us exactly that:
need to test the rendered output. [Vue][vue-test] guide's to unit test show us exactly that:
## Vue.js Expert Role
## Vue.js Expert Role
One should apply to be a Vue.js expert by opening an MR when the Merge Request's they create and review show:
One should apply to be a Vue.js expert by opening an MR when the Merge Request's they create and review show:
@@ -130,7 +130,7 @@ The following Elasticsearch settings are available:
...
@@ -130,7 +130,7 @@ The following Elasticsearch settings are available:
| `Elasticsearch indexing` | Enables/disables Elasticsearch indexing. You may want to enable indexing but disable search in order to give the index time to be fully completed, for example. Also, keep in mind that this option doesn't have any impact on existing data, this only enables/disables background indexer which tracks data changes. So by enabling this you will not get your existing data indexed, use special rake task for that as explained in [Adding GitLab's data to the Elasticsearch index](#adding-gitlabs-data-to-the-elasticsearch-index). |
| `Elasticsearch indexing` | Enables/disables Elasticsearch indexing. You may want to enable indexing but disable search in order to give the index time to be fully completed, for example. Also, keep in mind that this option doesn't have any impact on existing data, this only enables/disables background indexer which tracks data changes. So by enabling this you will not get your existing data indexed, use special rake task for that as explained in [Adding GitLab's data to the Elasticsearch index](#adding-gitlabs-data-to-the-elasticsearch-index). |
| `Use the new repository indexer (beta)` | Perform repository indexing using [GitLab Elasticsearch Indexer](https://gitlab.com/gitlab-org/gitlab-elasticsearch-indexer). |
| `Use the new repository indexer (beta)` | Perform repository indexing using [GitLab Elasticsearch Indexer](https://gitlab.com/gitlab-org/gitlab-elasticsearch-indexer). |
| `Search with Elasticsearch enabled` | Enables/disables using Elasticsearch in search. |
| `Search with Elasticsearch enabled` | Enables/disables using Elasticsearch in search. |
| `URL` | The URL to use for connecting to Elasticsearch. Use a comma-separated list to support clustering (e.g., "http://host1, https://host2:9200"). If your Elasticsearch instance is password protected, pass the `username:password` in the URL (e.g., `http://<username>:<password>@<elastic_host>:9200/`). |
| `URL` | The URL to use for connecting to Elasticsearch. Use a comma-separated list to support clustering (e.g., `http://host1, https://host2:9200`). If your Elasticsearch instance is password protected, pass the `username:password` in the URL (e.g., `http://<username>:<password>@<elastic_host>:9200/`). |
| `Number of Elasticsearch shards` | Elasticsearch indexes are split into multiple shards for performance reasons. In general, larger indexes need to have more shards. Changes to this value do not take effect until the index is recreated. You can read more about tradeoffs in the [Elasticsearch documentation](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-create-index.html#create-index-settings) |
| `Number of Elasticsearch shards` | Elasticsearch indexes are split into multiple shards for performance reasons. In general, larger indexes need to have more shards. Changes to this value do not take effect until the index is recreated. You can read more about tradeoffs in the [Elasticsearch documentation](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-create-index.html#create-index-settings) |
| `Number of Elasticsearch replicas` | Each Elasticsearch shard can have a number of replicas. These are a complete copy of the shard, and can provide increased query performance or resilience against hardware failure. Increasing this value will greatly increase total disk space required by the index. |
| `Number of Elasticsearch replicas` | Each Elasticsearch shard can have a number of replicas. These are a complete copy of the shard, and can provide increased query performance or resilience against hardware failure. Increasing this value will greatly increase total disk space required by the index. |
| `Limit namespaces and projects that can be indexed` | Enabling this will allow you to select namespaces and projects to index. All other namespaces and projects will use database search instead. Please note that if you enable this option but do not select any namespaces or projects, none will be indexed. [Read more below](#limiting-namespaces-and-projects).
| `Limit namespaces and projects that can be indexed` | Enabling this will allow you to select namespaces and projects to index. All other namespaces and projects will use database search instead. Please note that if you enable this option but do not select any namespaces or projects, none will be indexed. [Read more below](#limiting-namespaces-and-projects).
...
@@ -156,7 +156,7 @@ If no namespaces or projects are selected, no Elasticsearch indexing will take p
...
@@ -156,7 +156,7 @@ If no namespaces or projects are selected, no Elasticsearch indexing will take p
CAUTION: **Warning**:
CAUTION: **Warning**:
If you have already indexed your instance, you will have to regenerate the index in order to delete all existing data
If you have already indexed your instance, you will have to regenerate the index in order to delete all existing data
for filtering to work correctly. To do this run the rake tasks `gitlab:elastic:create_empty_index` and
for filtering to work correctly. To do this run the rake tasks `gitlab:elastic:create_empty_index` and
`gitlab:elastic:clear_index_status` Afterwards, removing a namespace or a projeect from the list will delete the data
`gitlab:elastic:clear_index_status`. Afterwards, removing a namespace or a project from the list will delete the data
from the Elasticsearch index as expected.
from the Elasticsearch index as expected.
## Disabling Elasticsearch
## Disabling Elasticsearch
...
@@ -304,7 +304,7 @@ For Elasticsearch 6.x, before proceeding with the force merge, the index should
...
@@ -304,7 +304,7 @@ For Elasticsearch 6.x, before proceeding with the force merge, the index should
```bash
```bash
curl --request PUT localhost:9200/gitlab-production/_settings --data'{
curl --request PUT localhost:9200/gitlab-production/_settings --data'{
"settings": {
"settings": {
"index.blocks.write": true
"index.blocks.write": true
} }'
} }'
```
```
...
@@ -319,7 +319,7 @@ After this, if your index is in read-only, switch back to read-write:
...
@@ -319,7 +319,7 @@ After this, if your index is in read-only, switch back to read-write:
```bash
```bash
curl --request PUT localhost:9200/gitlab-production/_settings --data'{
curl --request PUT localhost:9200/gitlab-production/_settings --data'{
"settings": {
"settings": {
"index.blocks.write": false
"index.blocks.write": false
} }'
} }'
```
```
...
@@ -392,9 +392,9 @@ When performing a search, the GitLab index will use the following scopes:
...
@@ -392,9 +392,9 @@ When performing a search, the GitLab index will use the following scopes:
Whenever a change or deletion is made to an indexed GitLab object (a merge request description is changed, a file is deleted from the master branch in a repository, a project is deleted, etc), a document in the index is deleted. However, since these are "soft" deletes, the overall number of "deleted documents", and therefore wasted space, increases. Elasticsearch does intelligent merging of segments in order to remove these deleted documents. However, depending on the amount and type of activity in your GitLab installation, it's possible to see as much as 50% wasted space in the index.
Whenever a change or deletion is made to an indexed GitLab object (a merge request description is changed, a file is deleted from the master branch in a repository, a project is deleted, etc), a document in the index is deleted. However, since these are "soft" deletes, the overall number of "deleted documents", and therefore wasted space, increases. Elasticsearch does intelligent merging of segments in order to remove these deleted documents. However, depending on the amount and type of activity in your GitLab installation, it's possible to see as much as 50% wasted space in the index.
In general, we recommend simply letting Elasticseach merge and reclaim space automatically, with the default settings. From [Lucene's Handling of Deleted Documents](https://www.elastic.co/blog/lucenes-handling-of-deleted-documents"Lucene's Handling of Deleted Documents"), _"Overall, besides perhaps decreasing the maximum segment size, it is best to leave Lucene's defaults as-is and not fret too much about when deletes are reclaimed."_
In general, we recommend simply letting Elasticsearch merge and reclaim space automatically, with the default settings. From [Lucene's Handling of Deleted Documents](https://www.elastic.co/blog/lucenes-handling-of-deleted-documents"Lucene's Handling of Deleted Documents"), _"Overall, besides perhaps decreasing the maximum segment size, it is best to leave Lucene's defaults as-is and not fret too much about when deletes are reclaimed."_
However, some larger installations may wish to tune the merge policy settings:
However, some larger installations may wish to tune the merge policy settings:
- Consider reducing the `index.merge.policy.max_merged_segment` size from the default 5 GB to maybe 2 GB or 3 GB. Merging only happens when a segment has at least 50% deletions. Smaller segment sizes will allow merging to happen more frequently.
- Consider reducing the `index.merge.policy.max_merged_segment` size from the default 5 GB to maybe 2 GB or 3 GB. Merging only happens when a segment has at least 50% deletions. Smaller segment sizes will allow merging to happen more frequently.
...
@@ -425,14 +425,14 @@ Here are some common pitfalls and how to overcome them:
...
@@ -425,14 +425,14 @@ Here are some common pitfalls and how to overcome them:
-**How can I verify my GitLab instance is using Elasticsearch?**
-**How can I verify my GitLab instance is using Elasticsearch?**
The easiest method is via the rails console (`sudo gitlab-rails console`) by running the following:
The easiest method is via the rails console (`sudo gitlab-rails console`) by running the following:
```ruby
```ruby
u = User.find_by_username('your-username')
u = User.find_by_username('your-username')
s = SearchService.new(u, {:search => 'search_term'})
s = SearchService.new(u, {:search => 'search_term'})
pp s.search_objects.class.name
pp s.search_objects.class.name
```
```
If you see `Elasticsearch::Model::Response::Records`, you are using Elasticsearch.
If you see `Elasticsearch::Model::Response::Records`, you are using Elasticsearch.
-**I updated GitLab and now I can't find anything**
-**I updated GitLab and now I can't find anything**
...
@@ -443,23 +443,23 @@ Here are some common pitfalls and how to overcome them:
...
@@ -443,23 +443,23 @@ Here are some common pitfalls and how to overcome them:
-**I indexed all the repositories but I can't find anything**
-**I indexed all the repositories but I can't find anything**
Make sure you indexed all the database data [as stated above](#adding-gitlabs-data-to-the-elasticsearch-index).
Make sure you indexed all the database data [as stated above](#adding-gitlabs-data-to-the-elasticsearch-index).
Beyond that, check via the [Elasticsearch Search API](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-search.html) to see if the data shows up on the Elasticsearch side.
Beyond that, check via the [Elasticsearch Search API](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-search.html) to see if the data shows up on the Elasticsearch side.
If it shows up via the [Elasticsearch Search API](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-search.html), check that it shows up via the rails console (`sudo gitlab-rails console`):
If it shows up via the [Elasticsearch Search API](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-search.html), check that it shows up via the rails console (`sudo gitlab-rails console`):
```ruby
```ruby
u = User.find_by_username('your-username')
u = User.find_by_username('your-username')
s = SearchService.new(u, {:search => 'search_term', :scope => ‘blobs’})
s = SearchService.new(u, {:search => 'search_term', :scope => ‘blobs’})
pp s.search_objects.to_a
pp s.search_objects.to_a
```
```
See [Elasticsearch Index Scopes](elasticsearch.md#elasticsearch-index-scopes) for more information on searching for specific types of data.
See [Elasticsearch Index Scopes](elasticsearch.md#elasticsearch-index-scopes) for more information on searching for specific types of data.
-**I indexed all the repositories but then switched Elasticsearch servers and now I can't find anything**
-**I indexed all the repositories but then switched Elasticsearch servers and now I can't find anything**
You will need to re-run all the rake tasks to re-index the database, repositories, and wikis.
You will need to re-run all the rake tasks to re-index the database, repositories, and wikis.
-**The indexing process is taking a very long time**
-**The indexing process is taking a very long time**
The more data present in your GitLab instance, the longer the indexing process takes.
The more data present in your GitLab instance, the longer the indexing process takes.
