Commit 6967bad3 authored by Douwe Maan's avatar Douwe Maan

Merge branch 'mirror-repository' into 'master'

Add option to mirror an upstream repository.

Closes internal https://dev.gitlab.org/gitlab/gitlab-ee/issues/279

Depends on gitlab-org/gitlab-shell!29

To do:

- [x] Decide on what user should be the author of the activity feed events. The initial project creator perhaps? That makes sense for personal projects, but less so for group projects, where it's even possible that the creator has since left the team.
- [x] Write specs!
- [x] Write documentation
- [x] Port back relevant commits to CE: gitlab-org/gitlab-ce!1822

----

## Mirror status on project homepage

![Screen_Shot_2015-11-12_at_12.48.15](/uploads/39666b0e4a14723407b0f66e2aaddb46/Screen_Shot_2015-11-12_at_12.48.15.png)


## Settings navigation item

![Screen_Shot_2015-11-10_at_17.32.27](/uploads/116ece85f65681bbd3c7c5e282ab3b3c/Screen_Shot_2015-11-10_at_17.32.27.png)

## Mirroring settings

![Screen_Shot_2015-11-17_at_11.17.17](/uploads/ff57c89831524435ec6a063c3c470203/Screen_Shot_2015-11-17_at_11.17.17.png)


## Activity feed with new "pushes" (mirror updates)
![activity_feed](/uploads/04754251f2142fd06b1fddc2aae45232/activity_feed.png)

I have yet to decide on what user should be the author of these events. The initial project creator perhaps? That makes sense for personal projects, but less so for group projects, where it's even possible that the creator has since left the team.

## Warning that branch has diverged from upstream
![diverged](/uploads/ecf7ae6c5709f626c750695f45ca521a/diverged.png)

## Commits heading after failed update
![commits_failure](/uploads/d1b3e1f6796431c839ee9e2f67499283/commits_failure.png)

Link send user to the mirroring settings if they are a project admin.

## Mirroring settings after failed update

![Screen_Shot_2015-11-12_at_12.43.09](/uploads/ece68d5021c9a4c374bf6d973cf9fafa/Screen_Shot_2015-11-12_at_12.43.09.png)

## New project page
![new_project](/uploads/296beb6c7faf390a9b7e1facc9c1d4d6/new_project.png)

## Import form after failed import
(Screenshot outdated)

![new_import_failure](/uploads/ac0ebfc8ec23232603126fe0a6e10171/new_import_failure.png)

Shown when the initial import of a project fails, not when an update of an existing project fails.

cc @dzaporozhets @sytses @JobV

See merge request !51
parents 1ab62227 19996fec
......@@ -8,6 +8,7 @@ v 8.2.0
- Fix "Rebase onto master"
- Ensure a comment is properly recorded in JIRA when a merge request is accepted
- Allow groups to appear in the `Share with group` share if the group owner allows it
- Add option to mirror an upstream repository.
v 8.1.4
- Fix bug in JIRA integration which prevented merge requests from being accepted when using issue closing pattern
......@@ -22,6 +23,7 @@ v 8.1.1
- Removed, see 8.1.2
v 8.1.0
- Add documentation for "Share project with group" API call
- Added an issues template (Hannes Rosenögger)
- Add documentation for "Share project with group" API call
- Abiliy to disable 'Share with Group' feature (via UI and API)
......@@ -69,7 +71,7 @@ v 7.14.0
- Fix importing projects from GitHub Enterprise Edition.
- Automatic approver suggestions (based on an authority of the code)
- Add support for Jenkins unstable status
- Automatic approver suggestions (based on an authority of the code)
- Automatic approver suggestions (based on an authority of the code)
- Support Kerberos ticket-based authentication for Git HTTP access
v 7.13.3
......
......@@ -102,6 +102,8 @@ class Dispatcher
new Activities()
when 'projects:group_links:index'
new GroupsSelect()
when 'projects:mirrors:show', 'projects:mirrors:update'
new UsersSelect()
when 'admin:emails:show'
new AdminEmailSelect()
......
......@@ -8,6 +8,7 @@ class @UsersSelect
@projectId = $(select).data('project-id')
@groupId = $(select).data('group-id')
@showCurrentUser = $(select).data('current-user')
@pushCodeToProtectedBranches = $(select).data('push-code-to-protected-branches')
showNullUser = $(select).data('null-user')
showAnyUser = $(select).data('any-user')
showEmailUser = $(select).data('email-user')
......@@ -112,6 +113,7 @@ class @UsersSelect
group_id: @groupId
skip_ldap: @skipLdap
current_user: @showCurrentUser
push_code_to_protected_branches: @pushCodeToProtectedBranches
dataType: "json"
).done (users) ->
callback(users)
......
......@@ -337,6 +337,10 @@ table {
}
}
.well {
margin-bottom: 0;
}
.search_box {
@extend .well;
text-align: center;
......
......@@ -172,7 +172,7 @@
}
.panel-body {
form {
form, pre {
margin: 0;
}
......
......@@ -106,13 +106,13 @@
.project-repo-buttons {
margin-top: 12px;
margin-bottom: 0px;
}
.btn {
@include btn-gray;
.btn {
@include btn-gray;
.count {
display: inline-block;
}
.count {
display: inline-block;
}
}
}
......@@ -205,6 +205,10 @@
.dropdown-toggle {
margin: -5px;
}
.update-mirror-button {
margin-right: -1px;
}
}
#notification-form {
......
......@@ -34,7 +34,11 @@ class AutocompleteController < ApplicationController
@users = @users.search(params[:search]) if params[:search].present?
@users = @users.active
@users = @users.reorder(:name)
@users = @users.page(params[:page]).per(PER_PAGE)
if params[:push_code_to_protected_branches] && project
@users = @users.to_a.select { |user| user.can?(:push_code_to_protected_branches, project) }.take(PER_PAGE)
else
@users = @users.page(params[:page]).per(PER_PAGE)
end
unless params[:search].present?
# Include current user if available to filter by "Me"
......
......@@ -8,9 +8,7 @@ class Projects::ImportsController < Projects::ApplicationController
end
def create
@project.import_url = params[:project][:import_url]
if @project.save
if @project.update_attributes(import_params)
@project.reload
if @project.import_failed?
......@@ -28,8 +26,8 @@ class Projects::ImportsController < Projects::ApplicationController
if @project.import_finished?
redirect_to(project_path(@project)) and return
else
redirect_to new_namespace_project_import_path(@project.namespace,
@project) && return
redirect_to(new_namespace_project_import_path(@project.namespace,
@project)) and return
end
end
end
......@@ -48,4 +46,8 @@ class Projects::ImportsController < Projects::ApplicationController
return
end
end
def import_params
params.require(:project).permit(:import_url, :mirror, :mirror_user_id)
end
end
class Projects::MirrorsController < Projects::ApplicationController
# Authorize
before_action :require_non_empty_project
before_action :authorize_admin_project!, except: [:update_now]
before_action :authorize_push_code!, only: [:update_now]
layout "project_settings"
def show
end
def update
if @project.update_attributes(mirror_params)
if @project.mirror?
@project.update_mirror
flash[:notice] = "Mirroring settings were successfully updated. The project is being updated."
elsif @project.mirror_changed?
flash[:notice] = "Mirroring was successfully disabled."
else
flash[:notice] = "Mirroring settings were successfully updated."
end
redirect_to namespace_project_mirror_path(@project.namespace, @project)
else
render :show
end
end
def update_now
@project.update_mirror
flash[:notice] = "The repository is being updated..."
redirect_back_or_default(default: namespace_project_path(@project.namespace, @project))
end
private
def mirror_params
params.require(:project).permit(:mirror, :import_url, :mirror_user_id)
end
end
......@@ -88,7 +88,8 @@ class ProjectsController < ApplicationController
end
def show
if @project.import_in_progress?
# If we're importing while we do have a repository, we're simply updating the mirror.
if @project.import_in_progress? && !@project.updating_mirror?
redirect_to namespace_project_import_path(@project.namespace, @project)
return
end
......@@ -235,6 +236,8 @@ class ProjectsController < ApplicationController
:merge_requests_ff_only_enabled,
:merge_requests_rebase_enabled,
:merge_requests_template,
:mirror,
:mirror_user_id,
:reset_approvals_on_push
)
end
......
......@@ -253,14 +253,6 @@ module ProjectsHelper
filename_path(project, :version)
end
def hidden_pass_url(original_url)
result = URI(original_url)
result.password = '*****' unless result.password.nil?
result
rescue
original_url
end
def project_wiki_path_with_version(proj, page, version, is_newest)
url_params = is_newest ? {} : { version_id: version }
namespace_project_wiki_path(proj.namespace, proj, page, url_params)
......
......@@ -13,6 +13,7 @@ module SelectsHelper
first_user = opts[:first_user] && current_user ? current_user.username : false
current_user = opts[:current_user] || false
project = opts[:project] || @project
push_code_to_protected_branches = opts[:push_code_to_protected_branches]
html = {
class: css_class,
......@@ -21,7 +22,8 @@ module SelectsHelper
'data-any-user' => any_user,
'data-email-user' => email_user,
'data-first-user' => first_user,
'data-current-user' => current_user
'data-current-user' => current_user,
'data-push-code-to-protected-branches' => push_code_to_protected_branches
}
unless opts[:scope] == :all
......
......@@ -76,6 +76,8 @@ class Project < ActiveRecord::Base
has_one :git_hook, dependent: :destroy
has_one :last_event, -> {order 'events.created_at DESC'}, class_name: 'Event', foreign_key: 'project_id'
belongs_to :mirror_user, foreign_key: 'mirror_user_id', class_name: 'User'
# Project services
has_many :services
has_one :gitlab_ci_service, dependent: :destroy
......@@ -161,6 +163,8 @@ class Project < ActiveRecord::Base
validates :import_url,
format: { with: /\A#{URI.regexp(%w(ssh git http https))}\z/, message: 'should be a valid url' },
if: :external_import?
validates :import_url, presence: true, if: :mirror?
validates :mirror_user, presence: true, if: :mirror?
validates :star_count, numericality: { greater_than_or_equal_to: 0 }
validate :check_limit, on: :create
validate :avatar_type,
......@@ -185,6 +189,7 @@ class Project < ActiveRecord::Base
scope :public_only, -> { where(visibility_level: Project::PUBLIC) }
scope :public_and_internal_only, -> { where(visibility_level: Project.public_and_internal_levels) }
scope :non_archived, -> { where(archived: false) }
scope :mirror, -> { where(mirror: true) }
state_machine :import_status, initial: :none do
event :import_start do
......@@ -209,6 +214,21 @@ class Project < ActiveRecord::Base
after_transition any => :started, do: :schedule_add_import_job
after_transition any => :finished, do: :clear_import_data
after_transition started: :finished do |project, transaction|
if project.mirror?
timestamp = DateTime.now
project.mirror_last_update_at = timestamp
project.mirror_last_successful_update_at = timestamp
project.save
end
end
after_transition started: :failed do |project, transaction|
if project.mirror?
project.update(mirror_last_update_at: DateTime.now)
end
end
end
class << self
......@@ -316,16 +336,26 @@ class Project < ActiveRecord::Base
end
def add_import_job
if forked?
unless RepositoryForkWorker.perform_async(id, forked_from_project.path_with_namespace, self.namespace.path)
import_fail
if repository_exists?
if mirror?
RepositoryUpdateMirrorWorker.perform_async(self.id)
end
return
end
if forked?
RepositoryForkWorker.perform_async(self.id, forked_from_project.path_with_namespace, self.namespace.path)
else
RepositoryImportWorker.perform_async(id)
RepositoryImportWorker.perform_async(self.id)
end
end
def clear_import_data
update(import_error: nil)
ProjectCacheWorker.perform_async(self.id)
self.import_data.destroy if self.import_data
end
......@@ -353,6 +383,62 @@ class Project < ActiveRecord::Base
import_status == 'finished'
end
def safe_import_url
result = URI.parse(self.import_url)
result.password = '*****' unless result.password.nil?
result.to_s
rescue
original_url
end
def mirror_updated?
mirror? && self.mirror_last_update_at
end
def updating_mirror?
mirror? && import_in_progress? && !empty_repo?
end
def mirror_last_update_status
return unless mirror_updated?
if self.mirror_last_update_at == self.mirror_last_successful_update_at
:success
else
:failed
end
end
def mirror_last_update_success?
mirror_last_update_status == :success
end
def mirror_last_update_failed?
mirror_last_update_status == :failed
end
def mirror_ever_updated_successfully?
mirror_updated? && self.mirror_last_successful_update_at
end
def update_mirror
return unless mirror?
return if import_in_progress?
if import_failed?
import_retry
else
import_start
end
end
def fetch_mirror
return unless mirror?
repository.fetch_upstream(self.import_url)
end
def check_limit
unless creator.can_create_project? or namespace.kind == 'group'
errors[:limit_reached] << ("Your project limit is #{creator.projects_limit} projects! Please contact your administrator to increase it")
......
......@@ -4,9 +4,11 @@ class Repository
class PreReceiveError < StandardError; end
class CommitError < StandardError; end
MIRROR_REMOTE = "upstream"
include Gitlab::ShellAdapter
attr_accessor :raw_repository, :path_with_namespace, :project
attr_accessor :path_with_namespace, :project
def self.clean_old_archives
repository_downloads_path = Gitlab.config.gitlab.repository_downloads_path
......@@ -19,14 +21,18 @@ class Repository
def initialize(path_with_namespace, default_branch = nil, project = nil)
@path_with_namespace = path_with_namespace
@project = project
end
if path_with_namespace
@raw_repository = Gitlab::Git::Repository.new(path_to_repo)
@raw_repository.autocrlf = :input
end
def raw_repository
return nil unless path_with_namespace
rescue Gitlab::Git::Repository::NoRepository
nil
@raw_repository ||= begin
repo = Gitlab::Git::Repository.new(path_to_repo)
repo.autocrlf = :input
repo
rescue Gitlab::Git::Repository::NoRepository
nil
end
end
# Return absolute path to repository
......@@ -105,37 +111,47 @@ class Repository
end
def add_branch(branch_name, ref)
cache.expire(:branch_names)
@branches = nil
expire_branches_cache
gitlab_shell.add_branch(path_with_namespace, branch_name, ref)
end
def add_tag(tag_name, ref, message = nil)
cache.expire(:tag_names)
@tags = nil
expire_tags_cache
gitlab_shell.add_tag(path_with_namespace, tag_name, ref, message)
end
def rm_branch(branch_name)
cache.expire(:branch_names)
@branches = nil
expire_branches_cache
gitlab_shell.rm_branch(path_with_namespace, branch_name)
end
def rm_tag(tag_name)
cache.expire(:tag_names)
@tags = nil
expire_tags_cache
gitlab_shell.rm_tag(path_with_namespace, tag_name)
end
def add_remote(name, url)
raw_repository.remote_add(name, url)
rescue Rugged::ConfigError
raw_repository.remote_update(name, url: url)
end
def fetch_remote(remote)
gitlab_shell.fetch_remote(path_with_namespace, remote)
end
def branch_names
cache.fetch(:branch_names) { raw_repository.branch_names }
end
def branch_exists?(name)
branch_names.include?(name)
end
def tag_names
cache.fetch(:tag_names) { raw_repository.tag_names }
end
......@@ -169,6 +185,16 @@ class Repository
end
end
def expire_tags_cache
cache.expire(:tag_names)
@tags = nil
end
def expire_branches_cache
cache.expire(:branch_names)
@branches = nil
end
def expire_cache
cache_keys.each do |key|
cache.expire(key)
......@@ -496,16 +522,49 @@ class Repository
root_ref_commit = commit(root_ref)
if branch_commit
rugged.merge_base(root_ref_commit.id, branch_commit.id) == branch_commit.id
is_ancestor?(branch_commit.id, root_ref_commit.id)
else
nil
end
end
def fetch_upstream(url)
add_remote(Repository::MIRROR_REMOTE, url)
fetch_remote(Repository::MIRROR_REMOTE)
end
def upstream_branches
rugged.references.each("refs/remotes/#{Repository::MIRROR_REMOTE}/*").map do |ref|
name = ref.name.sub(/\Arefs\/remotes\/#{Repository::MIRROR_REMOTE}\//, "")
begin
Gitlab::Git::Branch.new(name, ref.target)
rescue Rugged::ReferenceError
# Omit invalid branch
end
end.compact
end
def diverged_from_upstream?(branch_name)
branch_commit = commit(branch_name)
upstream_commit = commit("refs/remotes/#{MIRROR_REMOTE}/#{branch_name}")
if upstream_commit
!is_ancestor?(branch_commit.id, upstream_commit.id)
else
false
end
end
def merge_base(first_commit_id, second_commit_id)
rugged.merge_base(first_commit_id, second_commit_id)
end
def is_ancestor?(ancestor_id, descendant_id)
merge_base(ancestor_id, descendant_id) == ancestor_id
end
def search_files(query, ref)
offset = 2
args = %W(#{Gitlab.config.git.bin_path} grep -i -n --before-context #{offset} --after-context #{offset} -e #{query} #{ref || root_ref})
......
......@@ -55,7 +55,9 @@ module Projects
@project.save
if @project.persisted? && !@project.import?
raise 'Failed to create repository' unless @project.create_repository
unless @project.create_repository
raise 'Failed to create repository'
end
end
end
......@@ -94,9 +96,7 @@ module Projects
@project.team << [current_user, :master, current_user]
end
if @project.import?
@project.import_start
end
@project.import_start if @project.import?
predefined_git_hook = GitHook.find_by(is_sample: true)
......
module Projects
class UpdateMirrorService < BaseService
class Error < StandardError; end
class UpdateError < Error; end
def execute
return false unless project.mirror?
unless current_user.can?(:push_code_to_protected_branches, project)
return error("The mirror user is not allowed to push code to all branches on this project.")
end
update_tags do
project.fetch_mirror
end
update_branches
success
rescue Gitlab::Shell::Error, UpdateError => e
error(e.message)
end
private
def update_branches
local_branches = repository.branches.each_with_object({}) { |branch, branches| branches[branch.name] = branch }
repository.upstream_branches.each do |upstream_branch|
name = upstream_branch.name
local_branch = local_branches[name]
if local_branch.nil?
result = CreateBranchService.new(project, current_user).execute(name, upstream_branch.target)
if result[:status] == :error
raise UpdateError, result[:message]
end
elsif local_branch.target == upstream_branch.target
# Already up to date
elsif repository.diverged_from_upstream?(name)
# Cannot be updated
else
begin
repository.ff_merge(current_user, upstream_branch.target, name)
rescue Repository::PreReceiveError, Repository::CommitError => e
raise UpdateError, e.message
end
end
end
end
def update_tags(&block)
old_tags = repository.tags.each_with_object({}) { |tag, tags| tags[tag.name] = tag }
fetch_result = block.call
return fetch_result unless fetch_result
repository.expire_tags_cache
tags = repository.tags
tags.each do |tag|
old_tag = old_tags[tag.name]
old_tag_target = old_tag ? old_tag.target : Gitlab::Git::BLANK_SHA
next if old_tag_target == tag.target
GitTagPushService.new.execute(project, current_user, old_tag_target, tag.target, "#{Gitlab::Git::TAG_REF_PREFIX}#{tag.name}")
end
fetch_result
end
end
end
......@@ -44,6 +44,11 @@
= icon('lock fw')
%span
Protected Branches
= nav_link(controller: :mirrors) do
= link_to namespace_project_mirror_path(@project.namespace, @project), title: 'Mirror Repository', data: {placement: 'right'} do
= icon('clone fw')
%span
Mirror Repository
= nav_link(controller: :audit_events) do
= link_to namespace_project_audit_events_path(@project.namespace, @project) do
= icon('file-text-o fw')
......
......@@ -13,22 +13,24 @@
= link_to project_path(forked_from_project) do
= forked_from_project.namespace.try(:name)
- if @project.mirror?
- import_url = @project.safe_import_url
%p
Mirrored from #{link_to import_url, import_url}.
= render "shared/mirror_status"
.project-repo-buttons
.split-one
= render 'projects/buttons/star'
= render 'projects/buttons/fork'
- unless empty_repo
= render 'projects/buttons/fork'
= render "shared/clone_panel"
.split-repo-buttons
- unless empty_repo
- if can? current_user, :download_code, @project
= link_to archive_namespace_project_repository_path(@project.namespace, @project, ref: @ref, format: 'zip'), class: 'btn', rel: 'nofollow' do
= icon('download fw')
.split-repo-buttons
= render "projects/buttons/update_mirror"
= render "projects/buttons/download"
= render 'projects/buttons/dropdown'
= render 'projects/buttons/notifications'
......@@ -3,17 +3,25 @@
%div
= link_to namespace_project_tree_path(@project.namespace, @project, branch.name) do
%strong.str-truncated= branch.name
&nbsp;
- if branch.name == @repository.root_ref
%span.label.label-primary default
- elsif @repository.merged_to_root_ref? branch.name
%span.label.label-info.has_tooltip(title="Merged into #{@repository.root_ref}")
merged
- if @project.protected_branch? branch.name
%span.label.label-success
%i.fa.fa-lock
protected
- if branch.name == @repository.root_ref
%span.label.label-primary default
- elsif @repository.merged_to_root_ref? branch.name
%span.label.label-info.has_tooltip(title="Merged into #{@repository.root_ref}")
merged
- if @project.protected_branch? branch.name
%span.label.label-success
= icon('lock')
protected
- if @project.mirror_ever_updated_successfully? && @repository.diverged_from_upstream?(branch.name)
- tooltip_message = "The branch could not be updated automatically because it has diverged from its upstream counterpart."
- tooltip_message << "<br>To discard the local changes and overwrite the branch with the upstream version, delete it here and choose 'Update Now' above." if can?(current_user, :push_code, @project)
%span.label.label-danger.has_tooltip{data: { html: "true", title: tooltip_message }}
= icon('exclamation-triangle')
diverged from upstream
.controls.hidden-xs
- if create_mr_button?(@repository.root_ref, branch.name)
= link_to create_mr_path(@repository.root_ref, branch.name), class: 'btn btn-grouped btn-xs' do
......
......@@ -5,7 +5,7 @@
.pull-right
- if can? current_user, :push_code, @project
= link_to new_namespace_project_branch_path(@project.namespace, @project), class: 'btn btn-create' do
%i.fa.fa-add-sign
= icon('plus')
New branch
&nbsp;
.dropdown.inline
......@@ -26,6 +26,9 @@
= sort_title_oldest_updated
.oneline
Protected branches can be managed in project settings
= render 'projects/commits/mirror_status'
- unless @branches.empty?
%ul.content-list.all-branches
- @branches.each do |branch|
......
- unless @project.empty_repo?
- if can? current_user, :download_code, @project
= link_to archive_namespace_project_repository_path(@project.namespace, @project, ref: @ref, format: 'zip'), class: 'btn has_tooltip', rel: 'nofollow', title: "Download ZIP" do
= icon('download')
......@@ -32,5 +32,3 @@
= link_to new_namespace_project_tag_path(@project.namespace, @project) do
= icon('tags fw')
New tag
- if current_user && can?(current_user, :fork_project, @project)
- if current_user.already_forked?(@project) && current_user.manageable_namespaces.size < 2
= link_to namespace_project_path(current_user, current_user.fork_of(@project)), title: 'Go to your fork', class: 'btn' do
= icon('code-fork fw')
Fork
%span.count
= @project.forks_count
- else
= link_to new_namespace_project_fork_path(@project.namespace, @project), title: "Fork project", class: 'btn' do
= icon('code-fork fw')
%span.count
= @project.forks_count
- unless @project.empty_repo?
- if current_user && can?(current_user, :fork_project, @project)
- if current_user.already_forked?(@project) && current_user.manageable_namespaces.size < 2
= link_to namespace_project_path(current_user, current_user.fork_of(@project)), title: 'Go to your fork', class: 'btn has_tooltip' do
= icon('code-fork fw')
Fork
%span.count
= @project.forks_count
- else
= link_to new_namespace_project_fork_path(@project.namespace, @project), title: "Fork project", class: 'btn has_tooltip' do
= icon('code-fork fw')
%span.count
= @project.forks_count
- if current_user
= link_to toggle_star_namespace_project_path(@project.namespace, @project), class: 'btn star-btn toggle-star', method: :post, remote: true do
= link_to toggle_star_namespace_project_path(@project.namespace, @project), class: 'btn star-btn toggle-star has_tooltip', method: :post, remote: true, title: "Star project" do
= icon('star fw')
%span.count
= @project.star_count
......
- if @project.mirror? && can?(current_user, :push_code, @project)
- size = nil unless defined?(size) && size
- if @project.updating_mirror?
%span.btn.disabled.update-mirror-button.has_tooltip{title: "Updating from upstream..."}
= icon('refresh')
- else
= link_to update_now_namespace_project_mirror_path(@project.namespace, @project), method: :post, class: "btn update-mirror-button has_tooltip", title: "Update from upstream" do
= icon('refresh')
- if @project.mirror?
.gray-content-block.second-block
.pull-right
= render "shared/mirror_update_button"
- import_url = @project.safe_import_url
.oneline
This project is mirrored from #{link_to import_url, import_url}.
= render "shared/mirror_status"
......@@ -24,6 +24,8 @@
%ul.breadcrumb.repo-breadcrumb
= commits_breadcrumbs
= render 'projects/commits/mirror_status'
%div{id: dom_id(@project)}
#commits-list= render "commits", project: @project
.clear
......
- page_title "Import repository"
%h3.page-title
- if @project.import_failed?
Import failed. Retry?
- else
Import repository
Import repository
%hr
- if @project.import_failed?
.panel.panel-danger
.panel-heading The repository could not be imported.
.panel-body
%pre
:preserve
#{@project.import_error.try(:strip)}
= form_for @project, url: namespace_project_import_path(@project.namespace, @project), method: :post, html: { class: 'form-horizontal' } do |f|
.form-group.import-url-data
= f.label :import_url, class: 'control-label' do
%span Import existing git repo
.col-sm-10
= f.text_field :import_url, class: 'form-control', placeholder: 'https://github.com/randx/six.git'
.well.prepend-top-20
This URL must be publicly accessible or you can add a username and password like this: https://username:password@gitlab.com/company/project.git.
%br
The import will time out after 4 minutes. For big repositories, use a clone/push combination.
For SVN repositories, check #{link_to "this migrating from SVN doc.", "http://doc.gitlab.com/ce/workflow/importing/migrating_from_svn.html"}
= render "shared/import_form", f: f
.form-actions
= f.submit 'Start import', class: "btn btn-create", tabindex: 4
......@@ -8,7 +8,7 @@
- else
Import in progress.
- unless @project.forked?
%p.monospace git clone --bare #{hidden_pass_url(@project.import_url)}
%p.monospace git clone --bare #{@project.safe_import_url}
%p Please wait while we import the repository for you. Refresh at will.
:javascript
new ProjectImport();
- page_title "Mirror Repository"
.pull-right
- if @project.mirror_last_update_success?
Successfully updated #{time_ago_with_tooltip(@project.mirror_last_successful_update_at)}.
%span.prepend-left-10
= render "shared/mirror_update_button"
%h3.page-title
Mirror Repository
%p.light
Set up your project to automatically have its branches, tags, and commits updated from an upstream repository every hour.
%hr.clearfix
- if @project.mirror_last_update_failed?
.panel.panel-danger
.panel-heading
The repository failed to update #{time_ago_with_tooltip(@project.mirror_last_update_at)}.
- if @project.mirror_ever_updated_successfully?
Last successful update #{time_ago_with_tooltip(@project.mirror_last_successful_update_at)}.
.panel-body
%pre
:preserve
#{@project.import_error.try(:strip)}
= form_for @project, url: namespace_project_mirror_path(@project.namespace, @project), html: { class: 'form-horizontal' } do |f|
- if @project.errors.any?
.alert.alert-danger
- @project.errors.full_messages.each do |msg|
%p= msg
.form-group
.col-sm-offset-2.col-sm-10
.checkbox
= f.label :mirror do
= f.check_box :mirror
%strong
Mirror repository
.help-block
Automatically update this project's branches, tags, and commits from the upstream repository every hour.
.form-group
= f.label :import_url, class: 'control-label' do
%span Git repository URL
.col-sm-10
= f.text_field :import_url, class: 'form-control', placeholder: 'https://username:password@gitlab.company.com/group/project.git'
.well.prepend-top-20
%ul
%li
The repository must be accessible over <code>http://</code>, <code>https://</code>, <code>ssh://</code> or <code>git://</code>.
%li
If your HTTP repository is not publicly accessible, add authentication information to the URL: <code>https://username:password@gitlab.company.com/group/project.git</code>.
%li
If your SSH repository is not publicly accessible, add the public SSH key of the GitLab server to the remote repository.
%li
The update action will time out after 10 minutes. For big repositories, use a clone/push combination.
.form-group
= f.label :mirror_user_id, class: 'control-label' do
Mirror user
.col-sm-10
= users_select_tag("project[mirror_user_id]", class: 'input-large', selected: @project.mirror_user_id || current_user.id,
first_user: true, current_user: true, push_code_to_protected_branches: true)
.help-block
This user will be the author of all events in the activity feed that are the result of an update,
like new branches being created or new commits being pushed to existing branches.
They need to have at least master access to this project.
.form-actions
= f.submit "Save Changes", class: "btn btn-create"
......@@ -23,7 +23,6 @@
= f.select :namespace_id, namespaces_options(params[:namespace_id] || :current_user), {}, {class: 'select2', tabindex: 2}
- if import_sources_enabled?
.project-import.js-toggle-container
.form-group
%label.control-label Import project from
......@@ -82,19 +81,7 @@
%span Any repo by URL
.js-toggle-content.hide
.form-group.import-url-data
= f.label :import_url, class: 'control-label' do
%span Git repository URL
.col-sm-10
= f.text_field :import_url, class: 'form-control', placeholder: 'https://username:password@gitlab.company.com/group/project.git'
.well.prepend-top-20
%ul
%li
The repository must be accessible over HTTP(S). If it is not publicly accessible, you can add authentication information to the URL: <code>https://username:password@gitlab.company.com/group/project.git</code>.
%li
The import will time out after 4 minutes. For big repositories, use a clone/push combination.
%li
To migrate an SVN repository, check out #{link_to "this document", "http://doc.gitlab.com/ce/workflow/importing/migrating_from_svn.html"}.
= render "shared/import_form", f: f
.prepend-botton-10
......
......@@ -6,11 +6,13 @@
- if can? current_user, :push_code, @project
.pull-right
= link_to new_namespace_project_tag_path(@project.namespace, @project), class: 'btn btn-create new-tag-btn' do
%i.fa.fa-add-sign
= icon('plus')
New tag
.oneline
Tags give the ability to mark specific points in history as being important
= render 'projects/commits/mirror_status'
.tags
- unless @tags.empty?
%ul.content-list
......
......@@ -21,7 +21,6 @@
= gitlab_config.protocol.upcase
= text_field_tag :project_clone, default_url_to_repo(project), class: "js-select-on-focus form-control", readonly: true
- if project.kind_of?(Project)
.input-group-addon
.visibility-level-label.has_tooltip{'data-title' => "#{visibility_level_label(project.visibility_level)} project" }
.input-group-addon.has_tooltip{title: "#{visibility_level_label(project.visibility_level)} project", data: { container: "body" } }
.visibility-level-label
= visibility_level_icon(project.visibility_level)
.form-group.import-url-data
= f.label :import_url, class: 'control-label' do
%span Git repository URL
.col-sm-10
= f.text_field :import_url, class: 'form-control', placeholder: 'https://username:password@gitlab.company.com/group/project.git'
.well.prepend-top-20
%ul
%li
The repository must be accessible over <code>http://</code>, <code>https://</code> or <code>git://</code>.
%li
If your HTTP repository is not publicly accessible, add authentication information to the URL: <code>https://username:password@gitlab.company.com/group/project.git</code>.
%li
The import will time out after 4 minutes. For big repositories, use a clone/push combination.
%li
To migrate an SVN repository, check out #{link_to "this document", "http://doc.gitlab.com/ce/workflow/importing/migrating_from_svn.html"}.
.form-group
.col-sm-offset-2.col-sm-10
.checkbox
= f.label :mirror do
= f.check_box :mirror
%strong
Mirror repository
.help-block
Automatically update this project's branches and tags from the upstream repository every hour.
= f.hidden_field :mirror_user_id, value: current_user.id
- case @project.mirror_last_update_status
- when :success
Updated #{time_ago_with_tooltip(@project.mirror_last_successful_update_at)}.
- when :failed
- if can?(current_user, :admin_project, @project)
= link_to namespace_project_mirror_path(@project.namespace, @project) do
= icon("exclamation-triangle")
Failed to update #{time_ago_with_tooltip(@project.mirror_last_update_at)}.
- else
= icon("exclamation-triangle")
Failed to update #{time_ago_with_tooltip(@project.mirror_last_update_at)}.
- if @project.mirror_ever_updated_successfully?
Last successful update #{time_ago_with_tooltip(@project.mirror_last_successful_update_at)}.
- if @project.mirror? && can?(current_user, :push_code, @project)
- if @project.updating_mirror?
%span.btn.disabled
= icon('refresh')
Updating&hellip;
- else
= link_to update_now_namespace_project_mirror_path(@project.namespace, @project), method: :post, class: "btn" do
= icon('refresh')
Update Now
......@@ -13,22 +13,20 @@ class RepositoryForkWorker
end
result = gitlab_shell.fork_repository(source_path, target_path)
unless result
logger.error("Unable to fork project #{project_id} for repository #{source_path} -> #{target_path}")
project.update(import_error: "The project could not be forked.")
project.import_fail
project.save
return
end
if project.valid_repo?
ProjectCacheWorker.perform_async(project.id)
project.import_finish
else
project.import_fail
unless project.valid_repo?
logger.error("Project #{id} had an invalid repository after fork")
project.update(import_error: "The forked repository is invalid.")
project.import_fail
return
end
project.save
project.import_finish
end
end
......@@ -7,37 +7,52 @@ class RepositoryImportWorker
def perform(project_id)
project = Project.find(project_id)
unless project.import_url == Project::UNKNOWN_IMPORT_URL
import_result = gitlab_shell.send(:import_repository,
project.path_with_namespace,
project.import_url)
return project.import_fail unless import_result
else
if project.import_url == Project::UNKNOWN_IMPORT_URL
# In this case, we only want to import issues, not a repository.
unless project.create_repository
return project.import_fail
project.update(import_error: "The repository could not be created.")
project.import_fail
return
end
else
begin
gitlab_shell.import_repository(project.path_with_namespace, project.import_url)
rescue Gitlab::Shell::Error => e
project.update(import_error: e.message)
project.import_fail
return
end
end
data_import_result = case project.import_type
when 'github'
Gitlab::GithubImport::Importer.new(project).execute
when 'gitlab'
Gitlab::GitlabImport::Importer.new(project).execute
when 'bitbucket'
Gitlab::BitbucketImport::Importer.new(project).execute
when 'google_code'
Gitlab::GoogleCodeImport::Importer.new(project).execute
when 'fogbugz'
Gitlab::FogbugzImport::Importer.new(project).execute
else
true
end
return project.import_fail unless data_import_result
Gitlab::BitbucketImport::KeyDeleter.new(project).execute if project.import_type == 'bitbucket'
data_import_result =
case project.import_type
when 'github'
Gitlab::GithubImport::Importer.new(project).execute
when 'gitlab'
Gitlab::GitlabImport::Importer.new(project).execute
when 'bitbucket'
Gitlab::BitbucketImport::Importer.new(project).execute
when 'google_code'
Gitlab::GoogleCodeImport::Importer.new(project).execute
when 'fogbugz'
Gitlab::FogbugzImport::Importer.new(project).execute
else
true
end
unless data_import_result
project.update(import_error: "The remote issue data could not be imported.")
project.import_fail
return
end
if project.import_type == 'bitbucket'
Gitlab::BitbucketImport::KeyDeleter.new(project).execute
end
project.import_finish
project.save
ProjectCacheWorker.perform_async(project.id)
# Explicitly update mirror so that upstream remote is created and fetched
project.update_mirror
end
end
class RepositoryUpdateMirrorWorker
include Sidekiq::Worker
include Gitlab::ShellAdapter
sidekiq_options queue: :gitlab_shell
attr_accessor :project, :repository, :current_user
def perform(project_id)
@project = Project.find(project_id)
@current_user = @project.mirror_user || @project.creator
result = Projects::UpdateMirrorService.new(@project, @current_user).execute
if result[:status] == :error
project.update(import_error: result[:message])
project.import_fail
return
end
project.import_finish
end
end
class UpdateAllMirrorsWorker
include Sidekiq::Worker
include Sidetiq::Schedulable
recurrence { hourly }
def perform
Project.mirror.each(&:update_mirror)
end
end
......@@ -633,6 +633,11 @@ Gitlab::Application.routes.draw do
end
resources :protected_branches, only: [:index, :create, :update, :destroy], constraints: { id: Gitlab::Regex.git_reference_regex }
resource :mirror, only: [:show, :update] do
member do
post :update_now
end
end
resources :git_hooks, constraints: { id: /\d+/ }
resource :variables, only: [:show, :update]
resources :triggers, only: [:index, :create, :destroy]
......
class AddMirrorToProject < ActiveRecord::Migration
def change
add_column :projects, :mirror, :boolean, default: false, null: false
add_column :projects, :mirror_last_update_at, :datetime
add_column :projects, :mirror_last_successful_update_at, :datetime
add_column :projects, :mirror_user_id, :integer
end
end
class AddImportErrorToProject < ActiveRecord::Migration
def change
add_column :projects, :import_error, :text
end
end
......@@ -48,9 +48,9 @@ ActiveRecord::Schema.define(version: 20151114113410) do
t.boolean "twitter_sharing_enabled", default: true
t.text "help_text"
t.text "restricted_visibility_levels"
t.boolean "version_check_enabled", default: true
t.integer "max_attachment_size", default: 10, null: false
t.integer "default_project_visibility"
t.boolean "version_check_enabled", default: true
t.integer "default_snippet_visibility"
t.text "restricted_signup_domains"
t.boolean "user_oauth_applications", default: true
......@@ -700,31 +700,36 @@ ActiveRecord::Schema.define(version: 20151114113410) do
t.datetime "created_at"
t.datetime "updated_at"
t.integer "creator_id"
t.boolean "issues_enabled", default: true, null: false
t.boolean "wall_enabled", default: true, null: false
t.boolean "merge_requests_enabled", default: true, null: false
t.boolean "wiki_enabled", default: true, null: false
t.boolean "issues_enabled", default: true, null: false
t.boolean "wall_enabled", default: true, null: false
t.boolean "merge_requests_enabled", default: true, null: false
t.boolean "wiki_enabled", default: true, null: false
t.integer "namespace_id"
t.string "issues_tracker", default: "gitlab", null: false
t.string "issues_tracker", default: "gitlab", null: false
t.string "issues_tracker_id"
t.boolean "snippets_enabled", default: true, null: false
t.boolean "snippets_enabled", default: true, null: false
t.datetime "last_activity_at"
t.string "import_url"
t.integer "visibility_level", default: 0, null: false
t.boolean "archived", default: false, null: false
t.integer "visibility_level", default: 0, null: false
t.boolean "archived", default: false, null: false
t.string "avatar"
t.string "import_status"
t.float "repository_size", default: 0.0
t.text "merge_requests_template"
t.integer "star_count", default: 0, null: false
t.boolean "merge_requests_rebase_enabled", default: false
t.float "repository_size", default: 0.0
t.integer "star_count", default: 0, null: false
t.string "import_type"
t.string "import_source"
t.integer "approvals_before_merge", default: 0, null: false
t.boolean "reset_approvals_on_push", default: true
t.integer "commit_count", default: 0
t.boolean "merge_requests_ff_only_enabled", default: false
t.text "merge_requests_template"
t.boolean "merge_requests_rebase_enabled", default: false
t.integer "commit_count", default: 0
t.integer "approvals_before_merge", default: 0, null: false
t.boolean "reset_approvals_on_push", default: true
t.boolean "merge_requests_ff_only_enabled", default: false
t.text "issues_template"
t.text "import_error"
t.boolean "mirror", default: false, null: false
t.datetime "mirror_last_update_at"
t.datetime "mirror_last_successful_update_at"
t.integer "mirror_user_id"
end
add_index "projects", ["created_at", "id"], name: "index_projects_on_created_at_and_id", using: :btree
......
......@@ -24,3 +24,4 @@
- [Releases](releases.md)
- [Merge Requests](merge_requests.md)
- ["Work In Progress" Merge Requests](wip_merge_requests.md)
- [Repository Mirroring](repository_mirroring.md)
# Repository Mirroring
You can set up a project to automatically have its branches, tags, and commits updated from an upstream repository.
This is useful when a repository you're interested in is located on a different server, and you want to be able to browse its content and its activity using the familiar GitLab interface.
Mirror repositories are updated every hour, and all new branches, tags, and commits will be visible in the project's activity feed. Users with at least developer access to the project can also force an immediate update.
![Project page](repository_mirroring/project_page.png)
Since the repository on GitLab functions as a mirror of the upstream repository, you are advised not to push commits directly to the repository on GitLab. Instead, any commits should be pushed to the upstream repository, and will end up in the GitLab repository automatically within an hour, or when a forced update is initiated.
If you do manually update a branch in the GitLab repository, the branch will become diverged from upstream, and GitLab will no longer automatically update this branch to prevent any changes from being lost.
![Diverged branch](repository_mirroring/diverged_branch.png)
You can set up Repository Mirroring by navigating to Project Settings &gt; Repository Mirroring.
![Settings](repository_mirroring/settings.png)
......@@ -44,7 +44,6 @@ class Spinach::Features::NewProject < Spinach::FeatureSteps
git_import_instructions = first('.js-toggle-content')
expect(git_import_instructions).to be_visible
expect(git_import_instructions).to have_content "Git repository URL"
expect(git_import_instructions).to have_content "The repository must be accessible over HTTP(S). If it is not publicly accessible, you can add authentication information to the URL:"
end
step 'I click on "Google Code"' do
......
module Gitlab
class Shell
class AccessDenied < StandardError; end
class Error < StandardError; end
class KeyAdder < Struct.new(:io)
def add_key(id, key)
......@@ -36,8 +36,23 @@ module Gitlab
# import_repository("gitlab/gitlab-ci", "https://github.com/randx/six.git")
#
def import_repository(name, url)
Gitlab::Utils.system_silent([gitlab_shell_projects_path, 'import-project',
"#{name}.git", url, '240'])
output, status = Popen::popen([gitlab_shell_projects_path, 'import-project', "#{name}.git", url, '240'])
raise Error, output unless status.zero?
true
end
# Fetch remote for repository
#
# name - project path with namespace
# remote - remote name
#
# Ex.
# fetch_remote("gitlab/gitlab-ci", "upstream")
#
def fetch_remote(name, remote)
output, status = Popen::popen([gitlab_shell_projects_path, 'fetch-remote', "#{name}.git", remote, '600'])
raise Error, output unless status.zero?
true
end
# Move repository
......
......@@ -25,13 +25,6 @@ describe Projects::ForkService do
end
end
context 'fork project failure' do
it "fails due to transaction failure" do
@to_project = fork_project(@from_project, @to_user, false)
expect(@to_project.import_failed?)
end
end
context 'project already exists' do
it "should fail due to validation, not transaction failure" do
@existing_project = create(:project, creator_id: @to_user.id, name: @from_project.name, namespace: @to_namespace)
......@@ -66,7 +59,7 @@ describe Projects::ForkService do
context 'fork project for group' do
it 'group owner successfully forks project into the group' do
to_project = fork_project(@project, @group_owner, true, @opts)
to_project = fork_project(@project, @group_owner, @opts)
expect(to_project.owner).to eq(@group)
expect(to_project.namespace).to eq(@group)
expect(to_project.name).to eq(@project.name)
......@@ -78,7 +71,7 @@ describe Projects::ForkService do
context 'fork project for group when user not owner' do
it 'group developer should fail to fork project into the group' do
to_project = fork_project(@project, @developer, true, @opts)
to_project = fork_project(@project, @developer, @opts)
expect(to_project.errors[:namespace]).to eq(['is not valid'])
end
end
......@@ -87,7 +80,7 @@ describe Projects::ForkService do
it 'should fail due to validation, not transaction failure' do
existing_project = create(:project, name: @project.name,
namespace: @group)
to_project = fork_project(@project, @group_owner, true, @opts)
to_project = fork_project(@project, @group_owner, @opts)
expect(existing_project.persisted?).to be_truthy
expect(to_project.errors[:name]).to eq(['has already been taken'])
expect(to_project.errors[:path]).to eq(['has already been taken'])
......@@ -95,8 +88,8 @@ describe Projects::ForkService do
end
end
def fork_project(from_project, user, fork_success = true, params = {})
allow(RepositoryForkWorker).to receive(:perform_async).and_return(fork_success)
def fork_project(from_project, user, params = {})
allow(RepositoryForkWorker).to receive(:perform_async).and_return(true)
Projects::ForkService.new(from_project, user, params).execute
end
end
require 'spec_helper'
describe Projects::UpdateMirrorService do
let(:project) { create(:project) }
let(:repository) { project.repository }
let(:mirror_user) { project.owner }
subject { described_class.new(project, mirror_user) }
before do
project.import_url = Project::UNKNOWN_IMPORT_URL
project.mirror = true
project.mirror_user = mirror_user
project.save
end
describe "#execute" do
it "fetches the upstream repository" do
expect(project).to receive(:fetch_mirror)
subject.execute
end
it "succeeds" do
allow(project).to receive(:fetch_mirror) { fetch_mirror(repository) }
result = subject.execute
expect(result[:status]).to eq(:success)
end
describe "updating tags" do
it "creates new tags" do
allow(project).to receive(:fetch_mirror) { fetch_mirror(repository) }
subject.execute
expect(repository.tag_names).to include('new-tag')
end
end
describe "updating branches" do
it "creates new branches" do
allow(project).to receive(:fetch_mirror) { fetch_mirror(repository) }
subject.execute
expect(repository.branch_names).to include('new-branch')
end
it "updates existing branches" do
allow(project).to receive(:fetch_mirror) { fetch_mirror(repository) }
subject.execute
expect(repository.find_branch('existing-branch').target).to eq(repository.find_branch('master').target)
end
it "doesn't update diverged branches" do
allow(project).to receive(:fetch_mirror) { fetch_mirror(repository) }
subject.execute
expect(repository.find_branch('markdown').target).not_to eq(repository.find_branch('master').target)
end
end
describe "when the mirror user doesn't have access" do
let(:mirror_user) { create(:user) }
it "fails" do
allow(project).to receive(:fetch_mirror) { fetch_mirror(repository) }
result = subject.execute
expect(result[:status]).to eq(:error)
end
end
end
def fetch_mirror(repository)
rugged = repository.rugged
masterrev = repository.find_branch('master').target
parentrev = repository.commit(masterrev).parent_id
rugged.references.create('refs/heads/existing-branch', parentrev)
repository.expire_branches_cache
repository.branches
# New branch
rugged.references.create('refs/remotes/upstream/new-branch', masterrev)
# Updated existing branch
rugged.references.create('refs/remotes/upstream/existing-branch', masterrev)
# Diverged branch
rugged.references.create('refs/remotes/upstream/markdown', masterrev)
# New tag
rugged.references.create('refs/tags/new-tag', masterrev)
end
end
......@@ -12,7 +12,6 @@ describe RepositoryForkWorker do
project.path_with_namespace,
fork_project.namespace.path).
and_return(true)
expect(ProjectCacheWorker).to receive(:perform_async)
subject.perform(project.id,
project.path_with_namespace,
......
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment