API: Share projects only with groups current_user can access

Aims to address the issues here: https://gitlab.com/gitlab-org/gitlab-ce/issues/23004

* Projects can be shared with non-existent groups
* Projects can be shared with groups that the current user does not have access to read

Concerns:

The new implementation of the API endpoint allows projects to be shared with a larger range of groups than can be done via the web UI.

The form for sharing a project with a group uses the following API endpoint to index the available groups: https://gitlab.com/gitlab-org/gitlab-ce/blob/494269fc92f61098ee6bd635a0426129ce2c5456/lib/api/groups.rb#L17. The groups indexed in the web form will only be those groups that the user is currently a member of.

The new implementation allows projects to be shared with any group that the authenticated user has access to view. This widens the range of groups to those that are public and internal.

See merge request !2005
Signed-off-by: Rémy Coutable <remy@rymai.me>

Don't send Private-Token headers to Sentry

Fixes https://gitlab.com/gitlab-org/gitlab-ce/issues/22537

This bumps 'raven' (the Ruby gem we use to send errors to Sentry) to
version 2.0.2. We need 2.0.0 or newer to be able to sanitize HTTP
headers.

See merge request !2004
Signed-off-by: Rémy Coutable <remy@rymai.me>

Signed-off-by: Rémy Coutable <remy@rymai.me>

Escape HTML nodes in builds commands in ci linter

This MR removes call to `simple_format` that behaves like `String#html_safe`, thus it passes unescaped HTML tags to the view.

Closes #22541

See merge request !2001
Signed-off-by: Rémy Coutable <remy@rymai.me>

API: disable rails session auth for non-GET/HEAD requests

Closes https://gitlab.com/gitlab-org/gitlab-ce/issues/22435

See merge request !1999
Signed-off-by: Rémy Coutable <remy@rymai.me>

Set a restrictive CORS policy for the API

Closes https://gitlab.com/gitlab-org/gitlab-ce/issues/22450

See merge request !1998
Signed-off-by: Rémy Coutable <remy@rymai.me>

Enforce the fork_project permission in Projects::CreateService

Projects::ForkService delegates to this service almost entirely, but needed one small change so it would propagate create errors correctly.

CreateService#execute needs significant refactoring; it is now right at the complexity limit set by Rubocop. I avoided doing so in this commit to keep the diff as small as possible.

Closes https://gitlab.com/gitlab-org/gitlab-ce/issues/18028

See merge request !1996
Signed-off-by: Rémy Coutable <remy@rymai.me>

Allow the Rails cookie to be used for API authentication

Makes the Rails cookie into a valid authentication token for the Grape
API, and uses it instead of token authentication in frontend code that
uses the API.

Rendering the private token into client-side javascript is a security
risk; it may be stolen through XSS or other attacks. In general,
re-using API code in the frontend is more desirable than implementing
endless actions that return JSON.

Closes #18302

See merge request !1995
Signed-off-by: Rémy Coutable <remy@rymai.me>

[ci skip]
Signed-off-by: Rémy Coutable <remy@rymai.me>

Signed-off-by: Rémy Coutable <remy@rymai.me>

Exclude some pending or inactivated rows in Member scopes

An unapproved request or not-yet-accepted invite should not give access rights. Neither should a blocked user be considered a member of anything.

One visible outcome of this behaviour is that owners and masters of a group or project may be blocked, yet still receive notification emails for access requests.

Closes https://gitlab.com/gitlab-org/gitlab-ce/issues/21650

See merge request !1994
Signed-off-by: Rémy Coutable <remy@rymai.me>

lib/backup: fix broken permissions when creating repo dir

## What does this MR do?

This MR fixes an issue where gitlab:backup:restore will not work because the repositories directory is set up with bad permissions. The bad permissions will prevent access to the repositories by the git user, causing all kinds of troubles (e.g. gitlab-shell won't install hooks).

## Are there points in the code the reviewer needs to double check?

## Why was this MR needed?

Rake task gitlab:backup:restore would complete successfully, but gitlab:check will report errors and gitlab cannot access repos.

## Screenshots (if relevant)

* repositories.old.1472478726 was created without the fix
* repositories was created with the fix

![gitlab-repo-permissions](/uploads/6cf2de57fe4f1be0bd5b5e1a13234889/gitlab-repo-permissions.png)

## Does this MR meet the acceptance criteria?

- [x] [CHANGELOG](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/CHANGELOG) entry added
- [ ] [Documentation created/updated](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/development/doc_styleguide.md)
- [ ] API support added
- Tests
  - [ ] Added for this feature/bug
  - [ ] All builds are passing
- [ ] Conform by the [style guides](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/CONTRIBUTING.md#style-guides)
- [x] Branch has no merge conflicts with `master` (if you do - rebase it please)
- [x] [Squashed related commits together](https://git-scm.com/book/en/Git-Tools-Rewriting-History#Squashing-Commits)

## What are the relevant issue numbers?

See merge request !6098

Update doorkeeper to 4.2.0

Changelog: https://git.io/v6PnV

See merge request !5881
(cherry picked from commit c5aa31c8)

Update Hamlit to 2.6.1

Fixes gitlab-org/gitlab-ce#21025 and gitlab-org/gitlab-ce#21017

See merge request !5873
(cherry picked from commit e26ce27d)

Allow to sort by project size on admin projects page

Fixes #20317

See merge request !5797
(cherry picked from commit dd2088b0)

Set permissions to admin for importing a project via Import/Export

Fixes https://gitlab.com/gitlab-org/gitlab-ce/issues/20802

In order to import a project, it is now required to be an admin

Moved from https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/5766

See merge request !1983
(cherry picked from commit 966b3038)

Fix attribute inclusion import/export config ignored in some cases

In the `import_export.yml` file we define the inclusion of some of the attributes. For some reason, this isn't working in certain cases - very unfortunate this includes `user`. This has been introduced in 8.10.3.

Related https://gitlab.com/gitlab-org/gitlab-ce/issues/20802

See merge request !1982

[ci skip]

Upgrade Rails to 4.2.7.1 for security fixes.

Upgrades Rails from 4.2.7 to 4.2.7.1 for security fixes.

For more information: http://weblog.rubyonrails.org/2016/8/11/Rails-5-0-0-1-4-2-7-2-and-3-2-22-3-have-been-released/

This should be backported to all currently-supported releases.

See merge request !5781

Used cached value of project count to reduce DB load

Following !5746, backport a change from https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/637/diffs#diff-2.

## Does this MR meet the acceptance criteria?

- [x] [CHANGELOG](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/CHANGELOG) entry added
- Tests
  - [x] All builds are passing
- [x] Conform by the [style guides](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/CONTRIBUTING.md#style-guides)
- [x] Branch has no merge conflicts with `master` (if you do - rebase it please)
- [x] [Squashed related commits together](https://git-scm.com/book/en/Git-Tools-Rewriting-History#Squashing-Commits)

See merge request !5754
Signed-off-by: Rémy Coutable <remy@rymai.me>

Add a method in Project to return a cached value of total count of projects

This is in preparation to address the DB load caused by the counting
in gitlab-com/infrastructure#303.

See merge request !5746

Revert "Defend against 'Host' header injection"

We reverted this in omnibus but forgot to do it in the source NGINX template.

See merge request !5706
Signed-off-by: Rémy Coutable <remy@rymai.me>

Add a data migration to fix some missing timestamps in the members table (again)

Repeats an earlier migration to fix historic bad data in the members table (missing created_at and updated_at fields)

I'm expecting the WHERE clauses to be fast enough, and to return few enough rows, that the migration doesn't need to use batches, but I'm not too familiar with the size of these tables in the wild, so perhaps that's a poor assumption.

8.10 introduced a dependency on the  `members.created_at` field in the project and namespace member view. If bad data is present, viewing the list of members now results in an NoMethodError and a 500 response from GitLab. Although the previous migration should have fixed all bad rows, we have evidence that it didn't in at least one case, despite the migration claiming to have run in the past.

Closes #20568

See merge request !5670
Signed-off-by: Rémy Coutable <remy@rymai.me>

Signed-off-by: Rémy Coutable <remy@rymai.me>

Fix Import/Export error checking versions

Fixes small bug preventing the correct error message about Import/Export version being displayed.

Fixes https://gitlab.com/gitlab-org/gitlab-ce/issues/20536

- [x] [CHANGELOG](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/CHANGELOG) entry added
- Tests
  - [x] Added for this feature/bug
  - [x] All builds are passing
- [x] Conform by the [style guides](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/CONTRIBUTING.md#style-guides)
- [x] Branch has no merge conflicts with `master` (if you do - rebase it please)
- [x] [Squashed related commits together](https://git-scm.com/book/en/Git-Tools-Rewriting-History#Squashing-Commits)

See merge request !5638
Signed-off-by: Rémy Coutable <remy@rymai.me>

Fix Import/Export not working in HA mode

Use a shared path instead of `Tempfile` default `/tmp` so the import file is accessible by any GitLab instance.

Fixes https://gitlab.com/gitlab-org/gitlab-ce/issues/20506

See merge request !5618
Signed-off-by: Rémy Coutable <remy@rymai.me>

Signed-off-by: Rémy Coutable <remy@rymai.me>

Don’t close issues on original project

Closes #20527 -> https://gitlab.com/gitlab-org/gitlab-ce/issues/20527

See merge request !1981
Signed-off-by: Rémy Coutable <remy@rymai.me>