Merge branch 'master' of gitlab.com:gitlab-org/gitlab-ee into ee-psimyn-issue-note-refac

doc/administration/repository_storage_types.md

@@ -4,50 +4,63 @@
 
 ## Legacy Storage
 
-Legacy Storage is the storage behavior prior to version 10.0. For historical reasons, GitLab replicated the same
-mapping structure from the projects URLs:
+Legacy Storage is the storage behavior prior to version 10.0. For historical
+reasons, GitLab replicated the same mapping structure from the projects URLs:
 
- * Project's repository: `#{namespace}/#{project_name}.git`
- * Project's wiki: `#{namespace}/#{project_name}.wiki.git`
+* Project's repository: `#{namespace}/#{project_name}.git`
+* Project's wiki: `#{namespace}/#{project_name}.wiki.git`
 
-This structure made simple to migrate from existing solutions to GitLab and easy for Administrators to find where the
-repository is stored.
+This structure made it simple to migrate from existing solutions to GitLab and
+easy for Administrators to find where the repository is stored.
 
 On the other hand this has some drawbacks:
 
-Storage location will concentrate huge amount of top-level namespaces. The impact can be reduced by the introduction of [multiple storage paths][storage-paths].
+Storage location will concentrate huge amount of top-level namespaces. The
+impact can be reduced by the introduction of [multiple storage
+paths][storage-paths].
 
-Because Backups are a snapshot of the same URL mapping, if you try to recover a very old backup, you need to verify
-if any project has taken the place of an old removed project sharing the same URL. This means that `mygroup/myproject`
-from your backup may not be the same original project that is today in the same URL.
+Because backups are a snapshot of the same URL mapping, if you try to recover a
+very old backup, you need to verify whether any project has taken the place of
+an old removed or renamed project sharing the same URL. This means that
+`mygroup/myproject` from your backup may not be the same original project that
+is at that same URL today.
 
-Any change in the URL will need to be reflected on disk (when groups / users or projects are renamed). This can add a lot
-of load in big installations, and can be even worst if they are using any type of network based filesystem.
+Any change in the URL will need to be reflected on disk (when groups / users or
+projects are renamed). This can add a lot of load in big installations,
+especially if using any type of network based filesystem.
 
-Last, for GitLab Geo, this storage type means we have to synchronize the disk state, replicate renames in the correct
-order or we may end-up with wrong repository or missing data temporarily.
+For GitLab Geo in particular: Geo does work with legacy storage, but in some
+edge cases due to race conditions it can lead to errors when a project is
+renamed multiple times in short succession, or a project is deleted and
+recreated under the same name very quickly. We expect these race events to be
+rare, and we have not observed a race condition side-effect happening yet.
 
-This pattern also exists in other objects stored in GitLab, like issue Attachments, GitLab Pages artifacts,
-Docker Containers for the integrated Registry, etc.
+This pattern also exists in other objects stored in GitLab, like issue
+Attachments, GitLab Pages artifacts, Docker Containers for the integrated
+Registry, etc.
 
 ## Hashed Storage
 
-Hashed Storage is the new storage behavior we are rolling out with 10.0. It's not enabled by default yet, but we
-encourage everyone to try-it and take the time to fix any script you may have that depends on the old behavior.
+> **Warning:** Hashed storage is in **Alpha**. For the latest updates, check the
+> associated [issue](https://gitlab.com/gitlab-com/infrastructure/issues/2821)
+> and please report any problems you encounter.
 
-Instead of coupling project URL and the folder structure where the repository will be stored on disk, we are coupling
-a hash, based on the project's ID.
+Hashed Storage is the new storage behavior we are rolling out with 10.0. Instead
+of coupling project URL and the folder structure where the repository will be
+stored on disk, we are coupling a hash, based on the project's ID. This makes
+the folder structure immutable, and therefore eliminates any requirement to
+synchronize state from URLs to disk structure. This means that renaming a group,
+user, or project will cost only the database transaction, and will take effect
+immediately.
 
-This makes the folder structure immutable, and therefore eliminates any requirement to synchronize state from URLs to
-disk structure. This means that renaming a group, user or project will cost only the database transaction, and will take
-effect immediately.
+The hash also helps to spread the repositories more evenly on the disk, so the
+top-level directory will contain less folders than the total amount of top-level
+namespaces.
 
-The hash also helps to spread the repositories more evenly on the disk, so the top-level directory will contain less
-folders than the total amount of top-level namespaces.
-
-Hash format is based on hexadecimal representation of SHA256: `SHA256(project.id)`.
-Top-level folder uses first 2 characters, followed by another folder with the next 2 characters. They are both stored in
-a special folder `@hashed`, to co-exist with existing Legacy projects:
+The hash format is based on the hexadecimal representation of SHA256:
+`SHA256(project.id)`. The top-level folder uses the first 2 characters, followed
+by another folder with the next 2 characters. They are both stored in a special
+`@hashed` folder, to be able to co-exist with existing Legacy Storage projects:
 
 ```ruby
 # Project's repository:
@@ -57,15 +70,13 @@ a special folder `@hashed`, to co-exist with existing Legacy projects:
 "@hashed/#{hash[0..1]}/#{hash[2..3]}/#{hash}.wiki.git"
 ```
 
-This new format also makes possible to restore backups with confidence, as when restoring a repository from the backup,
-you will never mistakenly restore a repository in the wrong project (considering the backup is made after the migration).
-
 ### How to migrate to Hashed Storage
 
-In GitLab, go to **Admin > Settings**, find the **Repository Storage** section and select
-"_Create new projects using hashed storage paths_".
+In GitLab, go to **Admin > Settings**, find the **Repository Storage** section
+and select "_Create new projects using hashed storage paths_".
 
-To migrate your existing projects to the new storage type, check the specific [rake tasks].
+To migrate your existing projects to the new storage type, check the specific
+[rake tasks].
 
 [ce-28283]: https://gitlab.com/gitlab-org/gitlab-ce/issues/28283
 [rake tasks]: raketasks/storage.md#migrate-existing-projects-to-hashed-storage
@@ -73,11 +84,13 @@ To migrate your existing projects to the new storage type, check the specific [r
 
 ### Hashed Storage coverage
 
-We are incrementally moving every storable object in GitLab to the Hashed Storage pattern. You can check the current
-coverage status below.
+We are incrementally moving every storable object in GitLab to the Hashed
+Storage pattern. You can check the current coverage status below (and also see
+the [issue](https://gitlab.com/gitlab-com/infrastructure/issues/2821)).
 
-Note that things stored in an S3 compatible endpoint will not have the downsides mentioned earlier, if they are not
-prefixed with `#{namespace}/#{project_name}`, which is true for CI Cache and LFS Objects.
+Note that things stored in an S3 compatible endpoint will not have the downsides
+mentioned earlier, if they are not prefixed with `#{namespace}/#{project_name}`,
+which is true for CI Cache and LFS Objects.
 
 | Storable Object | Legacy Storage | Hashed Storage | S3 Compatible | GitLab Version |
 | --------------- | -------------- | -------------- | ------------- | -------------- |

doc/gitlab-geo/configuration.md

@@ -87,16 +87,12 @@ Meanwhile, the primary node will start to notify changes to the secondary, which
 will act on those notifications immediately. Make sure the secondary instance is
 running and accessible.
 
-### Step 2. Enabling hashed storage (from GitLab 10.0)
+### Step 2. Enabling hashed storage (optional, from GitLab 10.0)
 
 >**Warning**
-Hashed storage is in **Beta**. It is considered experimental and not
-production-ready. For the latest updates, check 
-[issue](https://gitlab.com/gitlab-com/infrastructure/issues/2821).
-Hashed Storage is not required to run GitLab Geo, but in some edge cases race
-conditions can lead to errors and Geo to break. Known issues are renaming a
-project multiple times in short succession, deleting a project and recreating
-with the same name very quickly.
+Hashed storage is in **Alpha**. It is considered experimental and not
+production-ready. See [Hashed
+Storage](../administration/repository_storage_types.md) for more detail.
 
 Using hashed storage significantly improves Geo replication - project and group
 renames no longer require synchronization between nodes.

doc/gitlab-geo/configuration_source.md

@@ -91,21 +91,12 @@ primary in a process known as backfill. Meanwhile, the primary node will start
 to notify changes to the secondary, which will act on those notifications
 immediately. Make sure the secondary instance is running and accessible.
 
-### Step 2. Enabling hashed storage (from GitLab 10.0)
+### Step 2. Enabling hashed storage (optional, GitLab 10.0)
 
->**Note:**
-Hashed storage is in **Beta**. It is considered experimental and not
-production-ready. For the latest updates, check 
-[issue](https://gitlab.com/gitlab-com/infrastructure/issues/2821).
-Hashed Storage is not required to run GitLab Geo, but in some edge cases race
-conditions can lead to errors and Geo to break. Known issues are renaming a
-project multiple times in short succession, deleting a project and recreating
-with the same name very quickly.
-
->**Note:**
-Instances already using hashed storage are not recommended to disable hashed
-storage, since bugs affecting hashed storage would continue to affect these
-projects.
+>**Warning**
+Hashed storage is in **Alpha**. It is considered experimental and not
+production-ready. See [Hashed
+Storage](../administration/repository_storage_types.md) for more detail.
 
 Using hashed storage significantly improves Geo replication - project and group
 renames no longer require synchronization between nodes.

doc/gitlab-geo/database.md

@@ -213,6 +213,10 @@ will not be able to perform all necessary configuration steps. Refer to
     postgresql['max_replication_slots'] = 1
     # postgresql['max_wal_senders'] = 10
     # postgresql['wal_keep_segments'] = 10
+
+    # Disable automatic database migrations for now
+    # (until PostgreSQL is restarted and listening on the interface address)
+    gitlab_rails['auto_migrate'] = false
     ```
 
     For external PostgreSQL instances, [see additional instructions][external postgresql].
@@ -228,27 +232,30 @@ will not be able to perform all necessary configuration steps. Refer to
     Replication documentation](https://www.postgresql.org/docs/9.6/static/runtime-config-replication.html)
     for more information.
 
-1. Save the file and [reconfigure GitLab][] for the database listen changes to
-   take effect.
-
-    **This step will fail.** This is caused by
-    [Omnibus#2797](https://gitlab.com/gitlab-org/omnibus-gitlab/issues/2797).
+1. Save the file and [reconfigure GitLab][] for the database listen changes and
+   the replication slot changes to be applied.
 
-    Restart PostgreSQL:
+    Restart PostgreSQL for its changes to take effect:
 
     ```bash
     gitlab-ctl restart postgresql
     ```
 
-    [Reconfigure GitLab][reconfigure GitLab] again. It should complete cleanly.
+1. Reenable migrations
+
+    Edit `/etc/gitlab/gitlab.rb` and **delete** the following lines:
+
+    ```ruby
+    # Disable automatic database migrations for now
+    # (until PostgreSQL is restarted and listening on the interface address)
+    gitlab_rails['auto_migrate'] = false
+    ```
 
-1. Restart your primary PostgreSQL server to ensure the replication slot
-   changes take effect (`sudo gitlab-ctl restart postgresql` for
-   Omnibus-provided PostgreSQL).
+    Save the file and [reconfigure GitLab][].
 
 1. Now that the PostgreSQL server is set up to accept remote connections, run
    `netstat -plnt` to make sure that PostgreSQL is listening on port `5432` to
-   the server's public IP.
+   the server's interface address.
 
 1. Verify that clock synchronization is enabled.
 

doc/gitlab-geo/updating_the_geo_nodes.md

@@ -53,18 +53,29 @@ secondary if ever promoted to a primary:
     sudo -u git -H rm ~git/.ssh/id_rsa ~git/.ssh/id_rsa.pub
     ```
 
+### Hashed Storage
+
+>**Warning**
+Hashed storage is in **Alpha**. It is considered experimental and not
+production-ready. See [Hashed
+Storage](../administration/repository_storage_types.md) for more detail.
+
+If you previously enabled Hashed Storage and migrated all your existing
+projects to Hashed Storage, disabling hashed storage will not migrate projects
+to their previous project based storage path. As such, once enabled and
+migrated we recommend leaving Hashed Storage enabled.
+
 ## Upgrading to GitLab 10.1
 
+>**Warning**
+Hashed storage is in **Alpha**. It is considered experimental and not
+production-ready. See [Hashed
+Storage](../administration/repository_storage_types.md) for more detail.
+
 [Hashed storage](../administration/repository_storage_types.md) was introduced
 in GitLab 10.0, and a [migration path](../administration/raketasks/storage.md)
 for existing repositories was added in GitLab 10.1.
 
-After upgrading to GitLab 10.1, we recommend that you
-[enable hashed storage for all new projects](#step-5-enabling-hashed-storage-from-gitlab-100),
-then [migrate existing projects to hashed storage](../administration/raketasks/storage.md).
-This will significantly reduce the amount of synchronization required between
-nodes in the event of project or group renames.
-
 ## Upgrading to GitLab 10.0
 
 Since GitLab 10.0, we require all **Geo** systems to [use SSH key lookups via

doc/user/project/issues/csv_export.md

@@ -22,7 +22,7 @@ Among numerous use cases for exporting issues for CSV, we can name a few:
 - Make a snapshot of issues for offline analysis or to communicate with other teams who may not be in GitLab
 - Create diagrams, graphs, and charts from the CSV data
 - Present the data in any other format for auditing or sharing reasons
-- Import the issues elsewhere
+- Import the issues elsewhere to a system outside of GitLab
 - Long-term issues' data analysis with multiple snapshots created along the time
 - Use the long-term data to gather relevant feedback given in the issues, and improve your product based on real metrics
 

spec/ee/spec/features/epics/epic_issues_spec.rb

@@ -43,6 +43,17 @@ describe 'Epic Issues', :js do
   end
 
   context 'when user is a group member' do
+    let(:issue_to_add) { create(:issue, project: private_project) }
+    let(:issue_invalid) { create(:issue) }
+
+    def add_issues(references)
+      find('.related-issues-block h3.panel-title button').click
+      find('.js-add-issuable-form-input').set(references)
+      find('.js-add-issuable-form-add-button').click
+
+      wait_for_requests
+    end
+
     before do
       group.add_developer(user)
       visit_epic
@@ -64,16 +75,20 @@ describe 'Epic Issues', :js do
       end
     end
 
+    it 'user cannot add new issues to the epic from another group' do
+      add_issues("#{issue_invalid.to_reference(full: true)}")
+
+      expect(page).to have_selector('.content-wrapper .alert-wrapper .flash-text')
+      expect(find('.flash-alert')).to have_text('No Issue found for given params')
+    end
+
     it 'user can add new issues to the epic' do
-      issue_to_add = create(:issue, project: private_project)
-      issue_invalid = create(:issue)
       references = "#{issue_to_add.to_reference(full: true)} #{issue_invalid.to_reference(full: true)}"
 
-      find('.related-issues-block h3.panel-title button').click
-      find('.js-add-issuable-form-input').set references
-      find('.js-add-issuable-form-add-button').click
+      add_issues(references)
 
-      wait_for_requests
+      expect(page).not_to have_selector('.content-wrapper .alert-wrapper .flash-text')
+      expect(page).not_to have_content('No Issue found for given params')
 
       within('.related-issues-block ul.issuable-list') do
         expect(page).to have_selector('li', count: 3)