Merge branch 'mirror-repository' into 'master'

Add option to mirror an upstream repository.

Closes internal https://dev.gitlab.org/gitlab/gitlab-ee/issues/279

Depends on gitlab-org/gitlab-shell!29

To do:

- [x] Decide on what user should be the author of the activity feed events. The initial project creator perhaps? That makes sense for personal projects, but less so for group projects, where it's even possible that the creator has since left the team.
- [x] Write specs!
- [x] Write documentation
- [x] Port back relevant commits to CE: gitlab-org/gitlab-ce!1822

----

## Mirror status on project homepage

![Screen_Shot_2015-11-12_at_12.48.15](/uploads/39666b0e4a14723407b0f66e2aaddb46/Screen_Shot_2015-11-12_at_12.48.15.png)


## Settings navigation item

![Screen_Shot_2015-11-10_at_17.32.27](/uploads/116ece85f65681bbd3c7c5e282ab3b3c/Screen_Shot_2015-11-10_at_17.32.27.png)

## Mirroring settings

![Screen_Shot_2015-11-17_at_11.17.17](/uploads/ff57c89831524435ec6a063c3c470203/Screen_Shot_2015-11-17_at_11.17.17.png)


## Activity feed with new "pushes" (mirror updates)
![activity_feed](/uploads/04754251f2142fd06b1fddc2aae45232/activity_feed.png)

I have yet to decide on what user should be the author of these events. The initial project creator perhaps? That makes sense for personal projects, but less so for group projects, where it's even possible that the creator has since left the team.

## Warning that branch has diverged from upstream
![diverged](/uploads/ecf7ae6c5709f626c750695f45ca521a/diverged.png)

## Commits heading after failed update
![commits_failure](/uploads/d1b3e1f6796431c839ee9e2f67499283/commits_failure.png)

Link send user to the mirroring settings if they are a project admin.

## Mirroring settings after failed update

![Screen_Shot_2015-11-12_at_12.43.09](/uploads/ece68d5021c9a4c374bf6d973cf9fafa/Screen_Shot_2015-11-12_at_12.43.09.png)

## New project page
![new_project](/uploads/296beb6c7faf390a9b7e1facc9c1d4d6/new_project.png)

## Import form after failed import
(Screenshot outdated)

![new_import_failure](/uploads/ac0ebfc8ec23232603126fe0a6e10171/new_import_failure.png)

Shown when the initial import of a project fails, not when an update of an existing project fails.

cc @dzaporozhets @sytses @JobV

See merge request !51

CHANGELOG-EE

@@ -8,6 +8,7 @@ v 8.2.0
   - Fix "Rebase onto master"
   - Ensure a comment is properly recorded in JIRA when a merge request is accepted
   - Allow groups to appear in the `Share with group` share if the group owner allows it
+  - Add option to mirror an upstream repository.
 
 v 8.1.4
   - Fix bug in JIRA integration which prevented merge requests from being accepted when using issue closing pattern
@@ -22,6 +23,7 @@ v 8.1.1
   - Removed, see 8.1.2
 
 v 8.1.0
+  - Add documentation for "Share project with group" API call
   - Added an issues template (Hannes Rosenögger)
   - Add documentation for "Share project with group" API call 
   - Abiliy to disable 'Share with Group' feature (via UI and API)
@@ -69,7 +71,7 @@ v 7.14.0
   - Fix importing projects from GitHub Enterprise Edition.
   - Automatic approver suggestions (based on an authority of the code)
   - Add support for Jenkins unstable status
-  - Automatic approver suggestions (based on an authority of the code)  
+  - Automatic approver suggestions (based on an authority of the code)
   - Support Kerberos ticket-based authentication for Git HTTP access
 
 v 7.13.3

app/assets/javascripts/dispatcher.js.coffee

@@ -102,6 +102,8 @@ class Dispatcher
         new Activities()
       when 'projects:group_links:index'
         new GroupsSelect()
+      when 'projects:mirrors:show', 'projects:mirrors:update'
+        new UsersSelect()
       when 'admin:emails:show'
         new AdminEmailSelect()
 

app/assets/javascripts/users_select.js.coffee

@@ -8,6 +8,7 @@ class @UsersSelect
       @projectId = $(select).data('project-id')
       @groupId = $(select).data('group-id')
       @showCurrentUser = $(select).data('current-user')
+      @pushCodeToProtectedBranches = $(select).data('push-code-to-protected-branches')
       showNullUser = $(select).data('null-user')
       showAnyUser = $(select).data('any-user')
       showEmailUser = $(select).data('email-user')
@@ -112,6 +113,7 @@ class @UsersSelect
         group_id: @groupId
         skip_ldap: @skipLdap
         current_user: @showCurrentUser
+        push_code_to_protected_branches: @pushCodeToProtectedBranches
       dataType: "json"
     ).done (users) ->
       callback(users)

app/assets/stylesheets/framework/common.scss

@@ -337,6 +337,10 @@ table {
   }
 }
 
+.well {
+  margin-bottom: 0;
+}
+
 .search_box {
   @extend .well;
   text-align: center;

app/assets/stylesheets/framework/tw_bootstrap.scss

@@ -172,7 +172,7 @@
   }
 
   .panel-body {
-    form {
+    form, pre {
       margin: 0;
     }
 

app/assets/stylesheets/pages/projects.scss

@@ -106,13 +106,13 @@
   .project-repo-buttons {
     margin-top: 12px;
     margin-bottom: 0px;
+  }
 
-    .btn {
-      @include btn-gray;
+  .btn {
+    @include btn-gray;
 
-      .count {
-        display: inline-block;
-      }
+    .count {
+      display: inline-block;
     }
   }
 }
@@ -205,6 +205,10 @@
   .dropdown-toggle  {
     margin: -5px;
   }
+
+  .update-mirror-button {
+    margin-right: -1px;
+  }
 }
 
 #notification-form {

app/controllers/autocomplete_controller.rb

@@ -34,7 +34,11 @@ class AutocompleteController < ApplicationController
     @users = @users.search(params[:search]) if params[:search].present?
     @users = @users.active
     @users = @users.reorder(:name)
-    @users = @users.page(params[:page]).per(PER_PAGE)
+    if params[:push_code_to_protected_branches] && project
+      @users = @users.to_a.select { |user| user.can?(:push_code_to_protected_branches, project) }.take(PER_PAGE)
+    else
+      @users = @users.page(params[:page]).per(PER_PAGE)
+    end
 
     unless params[:search].present?
       # Include current user if available to filter by "Me"

app/controllers/projects/imports_controller.rb

@@ -8,9 +8,7 @@ class Projects::ImportsController < Projects::ApplicationController
   end
 
   def create
-    @project.import_url = params[:project][:import_url]
-
-    if @project.save
+    if @project.update_attributes(import_params)
       @project.reload
 
       if @project.import_failed?
@@ -28,8 +26,8 @@ class Projects::ImportsController < Projects::ApplicationController
       if @project.import_finished?
         redirect_to(project_path(@project)) and return
       else
-        redirect_to new_namespace_project_import_path(@project.namespace,
-                                                      @project) && return
+        redirect_to(new_namespace_project_import_path(@project.namespace,
+                                                      @project)) and return
       end
     end
   end
@@ -48,4 +46,8 @@ class Projects::ImportsController < Projects::ApplicationController
         return
     end
   end
+
+  def import_params
+    params.require(:project).permit(:import_url, :mirror, :mirror_user_id)
+  end
 end

app/controllers/projects/mirrors_controller.rb

+class Projects::MirrorsController < Projects::ApplicationController
+  # Authorize
+  before_action :require_non_empty_project
+  before_action :authorize_admin_project!, except: [:update_now]
+  before_action :authorize_push_code!, only: [:update_now]
+
+  layout "project_settings"
+
+  def show
+
+  end
+
+  def update
+    if @project.update_attributes(mirror_params)
+      if @project.mirror?
+        @project.update_mirror
+
+        flash[:notice] = "Mirroring settings were successfully updated. The project is being updated."
+      elsif @project.mirror_changed?
+        flash[:notice] = "Mirroring was successfully disabled."
+      else
+        flash[:notice] = "Mirroring settings were successfully updated."
+      end
+
+      redirect_to namespace_project_mirror_path(@project.namespace, @project)
+    else
+      render :show
+    end
+  end
+
+  def update_now
+    @project.update_mirror
+
+    flash[:notice] = "The repository is being updated..."
+    redirect_back_or_default(default: namespace_project_path(@project.namespace, @project))
+  end
+
+  private
+
+  def mirror_params
+    params.require(:project).permit(:mirror, :import_url, :mirror_user_id)
+  end
+end

app/controllers/projects_controller.rb

@@ -88,7 +88,8 @@ class ProjectsController < ApplicationController
   end
 
   def show
-    if @project.import_in_progress?
+    # If we're importing while we do have a repository, we're simply updating the mirror.
+    if @project.import_in_progress? && !@project.updating_mirror?
       redirect_to namespace_project_import_path(@project.namespace, @project)
       return
     end
@@ -235,6 +236,8 @@ class ProjectsController < ApplicationController
       :merge_requests_ff_only_enabled,
       :merge_requests_rebase_enabled,
       :merge_requests_template,
+      :mirror,
+      :mirror_user_id,
       :reset_approvals_on_push
     )
   end

app/helpers/projects_helper.rb

@@ -253,14 +253,6 @@ module ProjectsHelper
     filename_path(project, :version)
   end
 
-  def hidden_pass_url(original_url)
-    result = URI(original_url)
-    result.password = '*****' unless result.password.nil?
-    result
-  rescue
-    original_url
-  end
-
   def project_wiki_path_with_version(proj, page, version, is_newest)
     url_params = is_newest ? {} : { version_id: version }
     namespace_project_wiki_path(proj.namespace, proj, page, url_params)

app/helpers/selects_helper.rb

@@ -13,6 +13,7 @@ module SelectsHelper
     first_user = opts[:first_user] && current_user ? current_user.username : false
     current_user = opts[:current_user] || false
     project = opts[:project] || @project
+    push_code_to_protected_branches = opts[:push_code_to_protected_branches]
 
     html = {
       class: css_class,
@@ -21,7 +22,8 @@ module SelectsHelper
       'data-any-user' => any_user,
       'data-email-user' => email_user,
       'data-first-user' => first_user,
-      'data-current-user' => current_user
+      'data-current-user' => current_user,
+      'data-push-code-to-protected-branches' => push_code_to_protected_branches
     }
 
     unless opts[:scope] == :all

app/models/project.rb

@@ -76,6 +76,8 @@ class Project < ActiveRecord::Base
   has_one :git_hook, dependent: :destroy
   has_one :last_event, -> {order 'events.created_at DESC'}, class_name: 'Event', foreign_key: 'project_id'
 
+  belongs_to :mirror_user, foreign_key: 'mirror_user_id', class_name: 'User'
+
   # Project services
   has_many :services
   has_one :gitlab_ci_service, dependent: :destroy
@@ -161,6 +163,8 @@ class Project < ActiveRecord::Base
   validates :import_url,
     format: { with: /\A#{URI.regexp(%w(ssh git http https))}\z/, message: 'should be a valid url' },
     if: :external_import?
+  validates :import_url, presence: true, if: :mirror?
+  validates :mirror_user, presence: true, if: :mirror?
   validates :star_count, numericality: { greater_than_or_equal_to: 0 }
   validate :check_limit, on: :create
   validate :avatar_type,
@@ -185,6 +189,7 @@ class Project < ActiveRecord::Base
   scope :public_only, -> { where(visibility_level: Project::PUBLIC) }
   scope :public_and_internal_only, -> { where(visibility_level: Project.public_and_internal_levels) }
   scope :non_archived, -> { where(archived: false) }
+  scope :mirror, -> { where(mirror: true) }
 
   state_machine :import_status, initial: :none do
     event :import_start do
@@ -209,6 +214,21 @@ class Project < ActiveRecord::Base
 
     after_transition any => :started, do: :schedule_add_import_job
     after_transition any => :finished, do: :clear_import_data
+
+    after_transition started: :finished do |project, transaction|
+      if project.mirror?
+        timestamp = DateTime.now
+        project.mirror_last_update_at = timestamp
+        project.mirror_last_successful_update_at = timestamp
+        project.save
+      end
+    end
+
+    after_transition started: :failed do |project, transaction|
+      if project.mirror?
+        project.update(mirror_last_update_at: DateTime.now)
+      end
+    end
   end
 
   class << self
@@ -316,16 +336,26 @@ class Project < ActiveRecord::Base
   end
 
   def add_import_job
-    if forked?
-      unless RepositoryForkWorker.perform_async(id, forked_from_project.path_with_namespace, self.namespace.path)
-        import_fail
+    if repository_exists?
+      if mirror?
+        RepositoryUpdateMirrorWorker.perform_async(self.id)
       end
+
+      return
+    end
+
+    if forked?
+      RepositoryForkWorker.perform_async(self.id, forked_from_project.path_with_namespace, self.namespace.path)
     else
-      RepositoryImportWorker.perform_async(id)
+      RepositoryImportWorker.perform_async(self.id)
     end
   end
 
   def clear_import_data
+    update(import_error: nil)
+
+    ProjectCacheWorker.perform_async(self.id)
+
     self.import_data.destroy if self.import_data
   end
 
@@ -353,6 +383,62 @@ class Project < ActiveRecord::Base
     import_status == 'finished'
   end
 
+  def safe_import_url
+    result = URI.parse(self.import_url)
+    result.password = '*****' unless result.password.nil?
+    result.to_s
+  rescue
+    original_url
+  end
+
+  def mirror_updated?
+    mirror? && self.mirror_last_update_at
+  end
+
+  def updating_mirror?
+    mirror? && import_in_progress? && !empty_repo?
+  end
+
+  def mirror_last_update_status
+    return unless mirror_updated?
+
+    if self.mirror_last_update_at == self.mirror_last_successful_update_at
+      :success
+    else
+      :failed
+    end
+  end
+
+  def mirror_last_update_success?
+    mirror_last_update_status == :success
+  end
+
+  def mirror_last_update_failed?
+    mirror_last_update_status == :failed
+  end
+
+  def mirror_ever_updated_successfully?
+    mirror_updated? && self.mirror_last_successful_update_at
+  end
+
+  def update_mirror
+    return unless mirror?
+
+    return if import_in_progress?
+
+    if import_failed?
+      import_retry
+    else
+      import_start
+    end
+  end
+
+  def fetch_mirror
+    return unless mirror?
+
+    repository.fetch_upstream(self.import_url)
+  end
+
   def check_limit
     unless creator.can_create_project? or namespace.kind == 'group'
       errors[:limit_reached] << ("Your project limit is #{creator.projects_limit} projects! Please contact your administrator to increase it")

app/models/repository.rb

@@ -4,9 +4,11 @@ class Repository
   class PreReceiveError < StandardError; end
   class CommitError < StandardError; end
 
+  MIRROR_REMOTE = "upstream"
+
   include Gitlab::ShellAdapter
 
-  attr_accessor :raw_repository, :path_with_namespace, :project
+  attr_accessor :path_with_namespace, :project
 
   def self.clean_old_archives
     repository_downloads_path = Gitlab.config.gitlab.repository_downloads_path
@@ -19,14 +21,18 @@ class Repository
   def initialize(path_with_namespace, default_branch = nil, project = nil)
     @path_with_namespace = path_with_namespace
     @project = project
+  end
 
-    if path_with_namespace
-      @raw_repository = Gitlab::Git::Repository.new(path_to_repo)
-      @raw_repository.autocrlf = :input
-    end
+  def raw_repository
+    return nil unless path_with_namespace
 
-  rescue Gitlab::Git::Repository::NoRepository
-    nil
+    @raw_repository ||= begin
+      repo = Gitlab::Git::Repository.new(path_to_repo)
+      repo.autocrlf = :input
+      repo
+    rescue Gitlab::Git::Repository::NoRepository
+      nil
+    end
   end
 
   # Return absolute path to repository
@@ -105,37 +111,47 @@ class Repository
   end
 
   def add_branch(branch_name, ref)
-    cache.expire(:branch_names)
-    @branches = nil
+    expire_branches_cache
 
     gitlab_shell.add_branch(path_with_namespace, branch_name, ref)
   end
 
   def add_tag(tag_name, ref, message = nil)
-    cache.expire(:tag_names)
-    @tags = nil
+    expire_tags_cache
 
     gitlab_shell.add_tag(path_with_namespace, tag_name, ref, message)
   end
 
   def rm_branch(branch_name)
-    cache.expire(:branch_names)
-    @branches = nil
+    expire_branches_cache
 
     gitlab_shell.rm_branch(path_with_namespace, branch_name)
   end
 
   def rm_tag(tag_name)
-    cache.expire(:tag_names)
-    @tags = nil
+    expire_tags_cache
 
     gitlab_shell.rm_tag(path_with_namespace, tag_name)
   end
 
+  def add_remote(name, url)
+    raw_repository.remote_add(name, url)
+  rescue Rugged::ConfigError
+    raw_repository.remote_update(name, url: url)
+  end
+
+  def fetch_remote(remote)
+    gitlab_shell.fetch_remote(path_with_namespace, remote)
+  end
+
   def branch_names
     cache.fetch(:branch_names) { raw_repository.branch_names }
   end
 
+  def branch_exists?(name)
+    branch_names.include?(name)
+  end
+
   def tag_names
     cache.fetch(:tag_names) { raw_repository.tag_names }
   end
@@ -169,6 +185,16 @@ class Repository
     end
   end
 
+  def expire_tags_cache
+    cache.expire(:tag_names)
+    @tags = nil
+  end
+
+  def expire_branches_cache
+    cache.expire(:branch_names)
+    @branches = nil
+  end
+
   def expire_cache
     cache_keys.each do |key|
       cache.expire(key)
@@ -496,16 +522,49 @@ class Repository
     root_ref_commit = commit(root_ref)
 
     if branch_commit
-      rugged.merge_base(root_ref_commit.id, branch_commit.id) == branch_commit.id
+      is_ancestor?(branch_commit.id, root_ref_commit.id)
     else
       nil
     end
   end
 
+  def fetch_upstream(url)
+    add_remote(Repository::MIRROR_REMOTE, url)
+    fetch_remote(Repository::MIRROR_REMOTE)
+  end
+
+  def upstream_branches
+    rugged.references.each("refs/remotes/#{Repository::MIRROR_REMOTE}/*").map do |ref|
+      name = ref.name.sub(/\Arefs\/remotes\/#{Repository::MIRROR_REMOTE}\//, "")
+
+      begin
+        Gitlab::Git::Branch.new(name, ref.target)
+      rescue Rugged::ReferenceError
+        # Omit invalid branch
+      end
+    end.compact
+  end
+
+  def diverged_from_upstream?(branch_name)
+    branch_commit = commit(branch_name)
+    upstream_commit = commit("refs/remotes/#{MIRROR_REMOTE}/#{branch_name}")
+
+    if upstream_commit
+      !is_ancestor?(branch_commit.id, upstream_commit.id)
+    else
+      false
+    end
+  end
+
   def merge_base(first_commit_id, second_commit_id)
     rugged.merge_base(first_commit_id, second_commit_id)
   end
 
+  def is_ancestor?(ancestor_id, descendant_id)
+    merge_base(ancestor_id, descendant_id) == ancestor_id
+  end
+
+
   def search_files(query, ref)
     offset = 2
     args = %W(#{Gitlab.config.git.bin_path} grep -i -n --before-context #{offset} --after-context #{offset} -e #{query} #{ref || root_ref})

app/services/projects/create_service.rb

@@ -55,7 +55,9 @@ module Projects
         @project.save
 
         if @project.persisted? && !@project.import?
-          raise 'Failed to create repository' unless @project.create_repository
+          unless @project.create_repository
+            raise 'Failed to create repository'
+          end
         end
       end
 
@@ -94,9 +96,7 @@ module Projects
         @project.team << [current_user, :master, current_user]
       end
 
-      if @project.import?
-        @project.import_start
-      end
+      @project.import_start if @project.import?
 
       predefined_git_hook = GitHook.find_by(is_sample: true)
 

app/services/projects/update_mirror_service.rb

+module Projects
+  class UpdateMirrorService < BaseService
+    class Error < StandardError; end
+    class UpdateError < Error; end
+
+    def execute
+      return false unless project.mirror?
+
+      unless current_user.can?(:push_code_to_protected_branches, project)
+        return error("The mirror user is not allowed to push code to all branches on this project.")
+      end
+
+      update_tags do
+        project.fetch_mirror
+      end
+
+      update_branches
+
+      success
+    rescue Gitlab::Shell::Error, UpdateError => e
+      error(e.message)
+    end
+
+    private
+
+    def update_branches
+      local_branches = repository.branches.each_with_object({}) { |branch, branches| branches[branch.name] = branch }
+
+      repository.upstream_branches.each do |upstream_branch|
+        name = upstream_branch.name
+
+        local_branch = local_branches[name]
+
+        if local_branch.nil?
+          result = CreateBranchService.new(project, current_user).execute(name, upstream_branch.target)
+          if result[:status] == :error
+            raise UpdateError, result[:message]
+          end
+        elsif local_branch.target == upstream_branch.target
+          # Already up to date
+        elsif repository.diverged_from_upstream?(name)
+          # Cannot be updated
+        else
+          begin
+            repository.ff_merge(current_user, upstream_branch.target, name)
+          rescue Repository::PreReceiveError, Repository::CommitError => e
+            raise UpdateError, e.message
+          end
+        end
+      end
+    end
+
+    def update_tags(&block)
+      old_tags = repository.tags.each_with_object({}) { |tag, tags| tags[tag.name] = tag }
+
+      fetch_result = block.call
+      return fetch_result unless fetch_result
+
+      repository.expire_tags_cache
+
+      tags = repository.tags
+
+      tags.each do |tag|
+        old_tag = old_tags[tag.name]
+        old_tag_target = old_tag ? old_tag.target : Gitlab::Git::BLANK_SHA
+
+        next if old_tag_target == tag.target
+
+        GitTagPushService.new.execute(project, current_user, old_tag_target, tag.target, "#{Gitlab::Git::TAG_REF_PREFIX}#{tag.name}")
+      end
+
+      fetch_result
+    end
+  end
+end

app/views/layouts/nav/_project_settings.html.haml

@@ -44,6 +44,11 @@
         = icon('lock fw')
         %span
           Protected Branches
+    = nav_link(controller: :mirrors) do
+      = link_to namespace_project_mirror_path(@project.namespace, @project), title: 'Mirror Repository', data: {placement: 'right'} do
+        = icon('clone fw')
+        %span
+          Mirror Repository
     = nav_link(controller: :audit_events) do
       = link_to namespace_project_audit_events_path(@project.namespace, @project) do
         = icon('file-text-o fw')

app/views/projects/_home_panel.html.haml

@@ -13,22 +13,24 @@
         = link_to project_path(forked_from_project) do
           = forked_from_project.namespace.try(:name)
 
+    - if @project.mirror?
+      - import_url = @project.safe_import_url
+      %p
+        Mirrored from #{link_to import_url, import_url}.
+
+        = render "shared/mirror_status"
 
 
   .project-repo-buttons
     .split-one
       = render 'projects/buttons/star'
+      = render 'projects/buttons/fork'
 
-      - unless empty_repo
-        = render 'projects/buttons/fork'
-    
     = render "shared/clone_panel"
-    .split-repo-buttons 
-      - unless empty_repo
-        - if can? current_user, :download_code, @project
-          = link_to archive_namespace_project_repository_path(@project.namespace, @project, ref: @ref, format: 'zip'), class: 'btn', rel: 'nofollow' do
-            = icon('download fw')
-            
+    
+    .split-repo-buttons
+      = render "projects/buttons/update_mirror"
+      = render "projects/buttons/download"
       = render 'projects/buttons/dropdown'
+
     = render 'projects/buttons/notifications'
-    

app/views/projects/branches/_branch.html.haml

@@ -3,17 +3,25 @@
   %div
     = link_to namespace_project_tree_path(@project.namespace, @project, branch.name) do
       %strong.str-truncated= branch.name
-       
-      - if branch.name == @repository.root_ref
-        %span.label.label-primary default
-      - elsif @repository.merged_to_root_ref? branch.name
-        %span.label.label-info.has_tooltip(title="Merged into #{@repository.root_ref}")
-          merged
 
-      - if @project.protected_branch? branch.name
-        %span.label.label-success
-          %i.fa.fa-lock
-          protected
+    - if branch.name == @repository.root_ref
+      %span.label.label-primary default
+    - elsif @repository.merged_to_root_ref? branch.name
+      %span.label.label-info.has_tooltip(title="Merged into #{@repository.root_ref}")
+        merged
+
+    - if @project.protected_branch? branch.name
+      %span.label.label-success
+        = icon('lock')
+        protected
+
+    - if @project.mirror_ever_updated_successfully? && @repository.diverged_from_upstream?(branch.name)
+      - tooltip_message = "The branch could not be updated automatically because it has diverged from its upstream counterpart."
+      - tooltip_message << "<br>To discard the local changes and overwrite the branch with the upstream version, delete it here and choose 'Update Now' above." if can?(current_user, :push_code, @project)
+      %span.label.label-danger.has_tooltip{data: { html: "true", title: tooltip_message }}
+        = icon('exclamation-triangle')
+        diverged from upstream
+
     .controls.hidden-xs
       - if create_mr_button?(@repository.root_ref, branch.name)
         = link_to create_mr_path(@repository.root_ref, branch.name), class: 'btn btn-grouped btn-xs' do

app/views/projects/branches/index.html.haml

@@ -5,7 +5,7 @@
   .pull-right
     - if can? current_user, :push_code, @project
       = link_to new_namespace_project_branch_path(@project.namespace, @project), class: 'btn btn-create' do
-        %i.fa.fa-add-sign
+        = icon('plus')
         New branch
        
     .dropdown.inline
@@ -26,6 +26,9 @@
             = sort_title_oldest_updated
   .oneline
     Protected branches can be managed in project settings
+
+= render 'projects/commits/mirror_status'
+
 - unless @branches.empty?
   %ul.content-list.all-branches
     - @branches.each do |branch|

app/views/projects/buttons/_download.html.haml

+- unless @project.empty_repo?
+  - if can? current_user, :download_code, @project
+    = link_to archive_namespace_project_repository_path(@project.namespace, @project, ref: @ref, format: 'zip'), class: 'btn has_tooltip', rel: 'nofollow', title: "Download ZIP" do
+      = icon('download')

app/views/projects/buttons/_dropdown.html.haml

@@ -32,5 +32,3 @@
           = link_to new_namespace_project_tag_path(@project.namespace, @project) do
             = icon('tags fw')
             New tag
-
-

app/views/projects/buttons/_fork.html.haml

-- if current_user && can?(current_user, :fork_project, @project)
-  - if current_user.already_forked?(@project) && current_user.manageable_namespaces.size < 2
-    = link_to namespace_project_path(current_user, current_user.fork_of(@project)), title: 'Go to your fork', class: 'btn' do
-      = icon('code-fork fw')
-      Fork
-      %span.count
-        = @project.forks_count
-  - else
-    = link_to new_namespace_project_fork_path(@project.namespace, @project), title: "Fork project", class: 'btn' do
-      = icon('code-fork fw')
-      %span.count
-        = @project.forks_count
+- unless @project.empty_repo?
+  - if current_user && can?(current_user, :fork_project, @project)
+    - if current_user.already_forked?(@project) && current_user.manageable_namespaces.size < 2
+      = link_to namespace_project_path(current_user, current_user.fork_of(@project)), title: 'Go to your fork', class: 'btn has_tooltip' do
+        = icon('code-fork fw')
+        Fork
+        %span.count
+          = @project.forks_count
+    - else
+      = link_to new_namespace_project_fork_path(@project.namespace, @project), title: "Fork project", class: 'btn has_tooltip' do
+        = icon('code-fork fw')
+        %span.count
+          = @project.forks_count

app/views/projects/buttons/_star.html.haml

 - if current_user
-  = link_to toggle_star_namespace_project_path(@project.namespace, @project), class: 'btn star-btn toggle-star', method: :post, remote: true do
+  = link_to toggle_star_namespace_project_path(@project.namespace, @project), class: 'btn star-btn toggle-star has_tooltip', method: :post, remote: true, title: "Star project" do
     = icon('star fw')
     %span.count
       = @project.star_count

app/views/projects/buttons/_update_mirror.html.haml

+- if @project.mirror? && can?(current_user, :push_code, @project)
+  - size = nil unless defined?(size) && size
+  - if @project.updating_mirror?
+    %span.btn.disabled.update-mirror-button.has_tooltip{title: "Updating from upstream..."}
+      = icon('refresh')
+  - else
+    = link_to update_now_namespace_project_mirror_path(@project.namespace, @project), method: :post, class: "btn update-mirror-button has_tooltip", title: "Update from upstream" do
+      = icon('refresh')

app/views/projects/commits/_mirror_status.html.haml

+- if @project.mirror?
+  .gray-content-block.second-block
+    .pull-right
+      = render "shared/mirror_update_button"
+
+    - import_url = @project.safe_import_url
+    .oneline
+      This project is mirrored from #{link_to import_url, import_url}.
+
+      = render "shared/mirror_status"

app/views/projects/commits/show.html.haml

@@ -24,6 +24,8 @@
   %ul.breadcrumb.repo-breadcrumb
     = commits_breadcrumbs
 
+= render 'projects/commits/mirror_status'
+
 %div{id: dom_id(@project)}
   #commits-list= render "commits", project: @project
 .clear

app/views/projects/imports/new.html.haml

 - page_title "Import repository"
 %h3.page-title
-  - if @project.import_failed?
-    Import failed. Retry?
-  - else
-    Import repository
+  Import repository
 
 %hr
 
+- if @project.import_failed?
+  .panel.panel-danger
+    .panel-heading The repository could not be imported.
+    .panel-body
+      %pre
+        :preserve
+          #{@project.import_error.try(:strip)}
+
 = form_for @project, url: namespace_project_import_path(@project.namespace, @project), method: :post, html: { class: 'form-horizontal' } do |f|
-  .form-group.import-url-data
-    = f.label :import_url, class: 'control-label' do
-      %span Import existing git repo
-    .col-sm-10
-      = f.text_field :import_url, class: 'form-control', placeholder: 'https://github.com/randx/six.git'
-      .well.prepend-top-20
-        This URL must be publicly accessible or you can add a username and password like this: https://username:password@gitlab.com/company/project.git.
-        %br
-        The import will time out after 4 minutes. For big repositories, use a clone/push combination.
-        For SVN repositories, check #{link_to "this migrating from SVN doc.", "http://doc.gitlab.com/ce/workflow/importing/migrating_from_svn.html"}
+  = render "shared/import_form", f: f
+
   .form-actions
     = f.submit 'Start import', class: "btn btn-create", tabindex: 4

app/views/projects/imports/show.html.haml

@@ -8,7 +8,7 @@
       - else
         Import in progress.
     - unless @project.forked?
-      %p.monospace git clone --bare #{hidden_pass_url(@project.import_url)}
+      %p.monospace git clone --bare #{@project.safe_import_url}
     %p Please wait while we import the repository for you. Refresh at will.
     :javascript
       new ProjectImport();

app/views/projects/mirrors/show.html.haml

+- page_title "Mirror Repository"
+.pull-right
+  - if @project.mirror_last_update_success?
+    Successfully updated #{time_ago_with_tooltip(@project.mirror_last_successful_update_at)}.
+  %span.prepend-left-10
+    = render "shared/mirror_update_button"
+%h3.page-title
+  Mirror Repository
+
+%p.light
+  Set up your project to automatically have its branches, tags, and commits updated from an upstream repository every hour.
+
+%hr.clearfix
+
+- if @project.mirror_last_update_failed?
+  .panel.panel-danger
+    .panel-heading
+      The repository failed to update #{time_ago_with_tooltip(@project.mirror_last_update_at)}.
+      - if @project.mirror_ever_updated_successfully?
+        Last successful update #{time_ago_with_tooltip(@project.mirror_last_successful_update_at)}.
+    .panel-body
+      %pre
+        :preserve
+          #{@project.import_error.try(:strip)}
+
+= form_for @project, url: namespace_project_mirror_path(@project.namespace, @project), html: { class: 'form-horizontal' } do |f|
+  - if @project.errors.any?
+    .alert.alert-danger
+      - @project.errors.full_messages.each do |msg|
+        %p= msg
+
+  .form-group
+    .col-sm-offset-2.col-sm-10
+      .checkbox
+        = f.label :mirror do
+          = f.check_box :mirror
+          %strong
+            Mirror repository
+        .help-block
+          Automatically update this project's branches, tags, and commits from the upstream repository every hour.
+
+  .form-group
+    = f.label :import_url, class: 'control-label' do
+      %span Git repository URL
+    .col-sm-10
+      = f.text_field :import_url, class: 'form-control', placeholder: 'https://username:password@gitlab.company.com/group/project.git'
+      .well.prepend-top-20
+        %ul
+          %li
+            The repository must be accessible over <code>http://</code>, <code>https://</code>, <code>ssh://</code> or <code>git://</code>.
+          %li
+            If your HTTP repository is not publicly accessible, add authentication information to the URL: <code>https://username:password@gitlab.company.com/group/project.git</code>.
+          %li
+            If your SSH repository is not publicly accessible, add the public SSH key of the GitLab server to the remote repository.
+          %li
+            The update action will time out after 10 minutes. For big repositories, use a clone/push combination.
+
+  .form-group
+    = f.label :mirror_user_id, class: 'control-label' do
+      Mirror user
+    .col-sm-10
+      = users_select_tag("project[mirror_user_id]", class: 'input-large', selected: @project.mirror_user_id || current_user.id,
+        first_user: true, current_user: true, push_code_to_protected_branches: true)
+      .help-block
+        This user will be the author of all events in the activity feed that are the result of an update,
+        like new branches being created or new commits being pushed to existing branches.
+        They need to have at least master access to this project.
+
+  .form-actions
+    = f.submit "Save Changes", class: "btn btn-create"

app/views/projects/new.html.haml

@@ -23,7 +23,6 @@
             = f.select :namespace_id, namespaces_options(params[:namespace_id] || :current_user), {}, {class: 'select2', tabindex: 2}
 
       - if import_sources_enabled?
-
         .project-import.js-toggle-container
           .form-group
             %label.control-label Import project from
@@ -82,19 +81,7 @@
                   %span Any repo by URL
 
           .js-toggle-content.hide
-            .form-group.import-url-data
-              = f.label :import_url, class: 'control-label' do
-                %span Git repository URL
-              .col-sm-10
-                = f.text_field :import_url, class: 'form-control', placeholder: 'https://username:password@gitlab.company.com/group/project.git'
-                .well.prepend-top-20
-                  %ul
-                    %li
-                      The repository must be accessible over HTTP(S). If it is not publicly accessible, you can add authentication information to the URL: <code>https://username:password@gitlab.company.com/group/project.git</code>.
-                    %li
-                      The import will time out after 4 minutes. For big repositories, use a clone/push combination.
-                    %li
-                      To migrate an SVN repository, check out #{link_to "this document", "http://doc.gitlab.com/ce/workflow/importing/migrating_from_svn.html"}.
+            = render "shared/import_form", f: f
 
       .prepend-botton-10
 

app/views/projects/tags/index.html.haml

@@ -6,11 +6,13 @@
   - if can? current_user, :push_code, @project
     .pull-right
       = link_to new_namespace_project_tag_path(@project.namespace, @project), class: 'btn btn-create new-tag-btn' do
-        %i.fa.fa-add-sign
+        = icon('plus')
         New tag
   .oneline
     Tags give the ability to mark specific points in history as being important
 
+= render 'projects/commits/mirror_status'
+
 .tags
   - unless @tags.empty?
     %ul.content-list

app/views/shared/_clone_panel.html.haml

@@ -21,7 +21,6 @@
         = gitlab_config.protocol.upcase
   = text_field_tag :project_clone, default_url_to_repo(project), class: "js-select-on-focus form-control", readonly: true
   - if project.kind_of?(Project)
-    .input-group-addon
-      .visibility-level-label.has_tooltip{'data-title' => "#{visibility_level_label(project.visibility_level)} project" }
+    .input-group-addon.has_tooltip{title: "#{visibility_level_label(project.visibility_level)} project", data: { container: "body" } }
+      .visibility-level-label
         = visibility_level_icon(project.visibility_level)
-   

app/views/shared/_import_form.html.haml

+.form-group.import-url-data
+  = f.label :import_url, class: 'control-label' do
+    %span Git repository URL
+  .col-sm-10
+    = f.text_field :import_url, class: 'form-control', placeholder: 'https://username:password@gitlab.company.com/group/project.git'
+
+    .well.prepend-top-20
+      %ul
+        %li
+          The repository must be accessible over <code>http://</code>, <code>https://</code> or <code>git://</code>.
+        %li
+          If your HTTP repository is not publicly accessible, add authentication information to the URL: <code>https://username:password@gitlab.company.com/group/project.git</code>.
+        %li
+          The import will time out after 4 minutes. For big repositories, use a clone/push combination.
+        %li
+          To migrate an SVN repository, check out #{link_to "this document", "http://doc.gitlab.com/ce/workflow/importing/migrating_from_svn.html"}.
+
+.form-group
+  .col-sm-offset-2.col-sm-10
+    .checkbox
+      = f.label :mirror do
+        = f.check_box :mirror
+        %strong
+          Mirror repository
+      .help-block
+        Automatically update this project's branches and tags from the upstream repository every hour.
+
+= f.hidden_field :mirror_user_id, value: current_user.id

app/views/shared/_mirror_status.html.haml

+- case @project.mirror_last_update_status
+- when :success
+  Updated #{time_ago_with_tooltip(@project.mirror_last_successful_update_at)}.
+- when :failed
+  - if can?(current_user, :admin_project, @project)
+    = link_to namespace_project_mirror_path(@project.namespace, @project) do
+      = icon("exclamation-triangle")
+      Failed to update #{time_ago_with_tooltip(@project.mirror_last_update_at)}.
+  - else
+    = icon("exclamation-triangle")
+    Failed to update #{time_ago_with_tooltip(@project.mirror_last_update_at)}.
+
+  - if @project.mirror_ever_updated_successfully?
+    Last successful update #{time_ago_with_tooltip(@project.mirror_last_successful_update_at)}.

app/views/shared/_mirror_update_button.html.haml

+- if @project.mirror? && can?(current_user, :push_code, @project)
+  - if @project.updating_mirror?
+    %span.btn.disabled
+      = icon('refresh')
+      Updating…
+  - else
+    = link_to update_now_namespace_project_mirror_path(@project.namespace, @project), method: :post, class: "btn" do
+      = icon('refresh')
+      Update Now

app/workers/repository_fork_worker.rb

@@ -13,22 +13,20 @@ class RepositoryForkWorker
     end
 
     result = gitlab_shell.fork_repository(source_path, target_path)
-
     unless result
       logger.error("Unable to fork project #{project_id} for repository #{source_path} -> #{target_path}")
+      project.update(import_error: "The project could not be forked.")
       project.import_fail
-      project.save
       return
     end
 
-    if project.valid_repo?
-      ProjectCacheWorker.perform_async(project.id)
-      project.import_finish
-    else
-      project.import_fail
+    unless project.valid_repo?
       logger.error("Project #{id} had an invalid repository after fork")
+      project.update(import_error: "The forked repository is invalid.")
+      project.import_fail
+      return
     end
 
-    project.save
+    project.import_finish
   end
 end

app/workers/repository_import_worker.rb

@@ -7,37 +7,52 @@ class RepositoryImportWorker
   def perform(project_id)
     project = Project.find(project_id)
 
-    unless project.import_url == Project::UNKNOWN_IMPORT_URL
-      import_result = gitlab_shell.send(:import_repository,
-                               project.path_with_namespace,
-                               project.import_url)
-      return project.import_fail unless import_result
-    else
+    if project.import_url == Project::UNKNOWN_IMPORT_URL
+      # In this case, we only want to import issues, not a repository.
       unless project.create_repository
-        return project.import_fail
+        project.update(import_error: "The repository could not be created.")
+        project.import_fail
+        return
+      end
+    else
+      begin
+        gitlab_shell.import_repository(project.path_with_namespace, project.import_url)
+      rescue Gitlab::Shell::Error => e
+        project.update(import_error: e.message)
+        project.import_fail
+        return
       end
     end
 
-    data_import_result = case project.import_type
-                         when 'github'
-                           Gitlab::GithubImport::Importer.new(project).execute
-                         when 'gitlab'
-                           Gitlab::GitlabImport::Importer.new(project).execute
-                         when 'bitbucket'
-                           Gitlab::BitbucketImport::Importer.new(project).execute
-                         when 'google_code'
-                           Gitlab::GoogleCodeImport::Importer.new(project).execute
-                         when 'fogbugz'
-                           Gitlab::FogbugzImport::Importer.new(project).execute
-                         else
-                           true
-                         end
-    return project.import_fail unless data_import_result
-
-    Gitlab::BitbucketImport::KeyDeleter.new(project).execute if project.import_type == 'bitbucket'
+    data_import_result =
+      case project.import_type
+      when 'github'
+        Gitlab::GithubImport::Importer.new(project).execute
+      when 'gitlab'
+        Gitlab::GitlabImport::Importer.new(project).execute
+      when 'bitbucket'
+        Gitlab::BitbucketImport::Importer.new(project).execute
+      when 'google_code'
+        Gitlab::GoogleCodeImport::Importer.new(project).execute
+      when 'fogbugz'
+        Gitlab::FogbugzImport::Importer.new(project).execute
+      else
+        true
+      end
+
+    unless data_import_result
+      project.update(import_error: "The remote issue data could not be imported.")
+      project.import_fail
+      return
+    end
+
+    if project.import_type == 'bitbucket'
+      Gitlab::BitbucketImport::KeyDeleter.new(project).execute
+    end
 
     project.import_finish
-    project.save
-    ProjectCacheWorker.perform_async(project.id)
+
+    # Explicitly update mirror so that upstream remote is created and fetched
+    project.update_mirror
   end
 end

app/workers/repository_update_mirror_worker.rb

+class RepositoryUpdateMirrorWorker
+  include Sidekiq::Worker
+  include Gitlab::ShellAdapter
+
+  sidekiq_options queue: :gitlab_shell
+
+  attr_accessor :project, :repository, :current_user
+
+  def perform(project_id)
+    @project = Project.find(project_id)
+    @current_user = @project.mirror_user || @project.creator
+
+    result = Projects::UpdateMirrorService.new(@project, @current_user).execute
+    if result[:status] == :error
+      project.update(import_error: result[:message])
+      project.import_fail
+      return
+    end
+
+    project.import_finish
+  end
+end

app/workers/update_all_mirrors_worker.rb

+class UpdateAllMirrorsWorker
+  include Sidekiq::Worker
+  include Sidetiq::Schedulable
+
+  recurrence { hourly }
+
+  def perform
+    Project.mirror.each(&:update_mirror)
+  end
+end

config/routes.rb

@@ -633,6 +633,11 @@ Gitlab::Application.routes.draw do
         end
 
         resources :protected_branches, only: [:index, :create, :update, :destroy], constraints: { id: Gitlab::Regex.git_reference_regex }
+        resource :mirror, only: [:show, :update] do
+          member do
+            post :update_now
+          end
+        end
         resources :git_hooks, constraints: { id: /\d+/ }
         resource :variables, only: [:show, :update]
         resources :triggers, only: [:index, :create, :destroy]

db/migrate/20151110125416_add_mirror_to_project.rb

+class AddMirrorToProject < ActiveRecord::Migration
+  def change
+    add_column :projects, :mirror, :boolean, default: false, null: false
+    add_column :projects, :mirror_last_update_at, :datetime
+    add_column :projects, :mirror_last_successful_update_at, :datetime
+    add_column :projects, :mirror_user_id, :integer
+  end
+end

db/migrate/20151110125604_add_import_error_to_project.rb

+class AddImportErrorToProject < ActiveRecord::Migration
+  def change
+    add_column :projects, :import_error, :text
+  end
+end

db/schema.rb

@@ -48,9 +48,9 @@ ActiveRecord::Schema.define(version: 20151114113410) do
     t.boolean  "twitter_sharing_enabled",      default: true
     t.text     "help_text"
     t.text     "restricted_visibility_levels"
-    t.boolean  "version_check_enabled",        default: true
     t.integer  "max_attachment_size",          default: 10,    null: false
     t.integer  "default_project_visibility"
+    t.boolean  "version_check_enabled",        default: true
     t.integer  "default_snippet_visibility"
     t.text     "restricted_signup_domains"
     t.boolean  "user_oauth_applications",      default: true
@@ -700,31 +700,36 @@ ActiveRecord::Schema.define(version: 20151114113410) do
     t.datetime "created_at"
     t.datetime "updated_at"
     t.integer  "creator_id"
-    t.boolean  "issues_enabled",                 default: true,     null: false
-    t.boolean  "wall_enabled",                   default: true,     null: false
-    t.boolean  "merge_requests_enabled",         default: true,     null: false
-    t.boolean  "wiki_enabled",                   default: true,     null: false
+    t.boolean  "issues_enabled",                   default: true,     null: false
+    t.boolean  "wall_enabled",                     default: true,     null: false
+    t.boolean  "merge_requests_enabled",           default: true,     null: false
+    t.boolean  "wiki_enabled",                     default: true,     null: false
     t.integer  "namespace_id"
-    t.string   "issues_tracker",                 default: "gitlab", null: false
+    t.string   "issues_tracker",                   default: "gitlab", null: false
     t.string   "issues_tracker_id"
-    t.boolean  "snippets_enabled",               default: true,     null: false
+    t.boolean  "snippets_enabled",                 default: true,     null: false
     t.datetime "last_activity_at"
     t.string   "import_url"
-    t.integer  "visibility_level",               default: 0,        null: false
-    t.boolean  "archived",                       default: false,    null: false
+    t.integer  "visibility_level",                 default: 0,        null: false
+    t.boolean  "archived",                         default: false,    null: false
     t.string   "avatar"
     t.string   "import_status"
-    t.float    "repository_size",                default: 0.0
-    t.text     "merge_requests_template"
-    t.integer  "star_count",                     default: 0,        null: false
-    t.boolean  "merge_requests_rebase_enabled",  default: false
+    t.float    "repository_size",                  default: 0.0
+    t.integer  "star_count",                       default: 0,        null: false
     t.string   "import_type"
     t.string   "import_source"
-    t.integer  "approvals_before_merge",         default: 0,        null: false
-    t.boolean  "reset_approvals_on_push",        default: true
-    t.integer  "commit_count",                   default: 0
-    t.boolean  "merge_requests_ff_only_enabled", default: false
+    t.text     "merge_requests_template"
+    t.boolean  "merge_requests_rebase_enabled",    default: false
+    t.integer  "commit_count",                     default: 0
+    t.integer  "approvals_before_merge",           default: 0,        null: false
+    t.boolean  "reset_approvals_on_push",          default: true
+    t.boolean  "merge_requests_ff_only_enabled",   default: false
     t.text     "issues_template"
+    t.text     "import_error"
+    t.boolean  "mirror",                           default: false,    null: false
+    t.datetime "mirror_last_update_at"
+    t.datetime "mirror_last_successful_update_at"
+    t.integer  "mirror_user_id"
   end
 
   add_index "projects", ["created_at", "id"], name: "index_projects_on_created_at_and_id", using: :btree

doc/workflow/README.md

@@ -24,3 +24,4 @@
 - [Releases](releases.md)
 - [Merge Requests](merge_requests.md)
 - ["Work In Progress" Merge Requests](wip_merge_requests.md)
+- [Repository Mirroring](repository_mirroring.md)

doc/workflow/repository_mirroring.md

+# Repository Mirroring
+
+You can set up a project to automatically have its branches, tags, and commits updated from an upstream repository.
+This is useful when a repository you're interested in is located on a different server, and you want to be able to browse its content and its activity using the familiar GitLab interface.
+
+Mirror repositories are updated every hour, and all new branches, tags, and commits will be visible in the project's activity feed. Users with at least developer access to the project can also force an immediate update.
+
+![Project page](repository_mirroring/project_page.png)
+
+Since the repository on GitLab functions as a mirror of the upstream repository, you are advised not to push commits directly to the repository on GitLab. Instead, any commits should be pushed to the upstream repository, and will end up in the GitLab repository automatically within an hour, or when a forced update is initiated.
+
+If you do manually update a branch in the GitLab repository, the branch will become diverged from upstream, and GitLab will no longer automatically update this branch to prevent any changes from being lost.
+
+![Diverged branch](repository_mirroring/diverged_branch.png)
+
+You can set up Repository Mirroring by navigating to Project Settings > Repository Mirroring.
+
+![Settings](repository_mirroring/settings.png)

features/steps/dashboard/new_project.rb

@@ -44,7 +44,6 @@ class Spinach::Features::NewProject < Spinach::FeatureSteps
     git_import_instructions = first('.js-toggle-content')
     expect(git_import_instructions).to be_visible
     expect(git_import_instructions).to have_content "Git repository URL"
-    expect(git_import_instructions).to have_content "The repository must be accessible over HTTP(S). If it is not publicly accessible, you can add authentication information to the URL:"
   end
 
   step 'I click on "Google Code"' do

lib/gitlab/backend/shell.rb

 module Gitlab
   class Shell
-    class AccessDenied < StandardError; end
+    class Error < StandardError; end
 
     class KeyAdder < Struct.new(:io)
       def add_key(id, key)
@@ -36,8 +36,23 @@ module Gitlab
     #   import_repository("gitlab/gitlab-ci", "https://github.com/randx/six.git")
     #
     def import_repository(name, url)
-      Gitlab::Utils.system_silent([gitlab_shell_projects_path, 'import-project',
-                                   "#{name}.git", url, '240'])
+      output, status = Popen::popen([gitlab_shell_projects_path, 'import-project', "#{name}.git", url, '240'])
+      raise Error, output unless status.zero?
+      true
+    end
+
+    # Fetch remote for repository
+    #
+    # name - project path with namespace
+    # remote - remote name
+    #
+    # Ex.
+    #   fetch_remote("gitlab/gitlab-ci", "upstream")
+    #
+    def fetch_remote(name, remote)
+      output, status = Popen::popen([gitlab_shell_projects_path, 'fetch-remote', "#{name}.git", remote, '600'])
+      raise Error, output unless status.zero?
+      true
     end
 
     # Move repository

spec/services/projects/fork_service_spec.rb

@@ -25,13 +25,6 @@ describe Projects::ForkService do
       end
     end
 
-    context 'fork project failure' do
-      it "fails due to transaction failure" do
-        @to_project = fork_project(@from_project, @to_user, false)
-        expect(@to_project.import_failed?)
-      end
-    end
-
     context 'project already exists' do
       it "should fail due to validation, not transaction failure" do
         @existing_project = create(:project, creator_id: @to_user.id, name: @from_project.name, namespace: @to_namespace)
@@ -66,7 +59,7 @@ describe Projects::ForkService do
 
     context 'fork project for group' do
       it 'group owner successfully forks project into the group' do
-        to_project = fork_project(@project, @group_owner, true, @opts)
+        to_project = fork_project(@project, @group_owner, @opts)
         expect(to_project.owner).to       eq(@group)
         expect(to_project.namespace).to   eq(@group)
         expect(to_project.name).to        eq(@project.name)
@@ -78,7 +71,7 @@ describe Projects::ForkService do
 
     context 'fork project for group when user not owner' do
       it 'group developer should fail to fork project into the group' do
-        to_project = fork_project(@project, @developer, true, @opts)
+        to_project = fork_project(@project, @developer, @opts)
         expect(to_project.errors[:namespace]).to eq(['is not valid'])
       end
     end
@@ -87,7 +80,7 @@ describe Projects::ForkService do
       it 'should fail due to validation, not transaction failure' do
         existing_project = create(:project, name: @project.name,
                                             namespace: @group)
-        to_project = fork_project(@project, @group_owner, true, @opts)
+        to_project = fork_project(@project, @group_owner, @opts)
         expect(existing_project.persisted?).to be_truthy
         expect(to_project.errors[:name]).to eq(['has already been taken'])
         expect(to_project.errors[:path]).to eq(['has already been taken'])
@@ -95,8 +88,8 @@ describe Projects::ForkService do
     end
   end
 
-  def fork_project(from_project, user, fork_success = true, params = {})
-    allow(RepositoryForkWorker).to receive(:perform_async).and_return(fork_success)
+  def fork_project(from_project, user, params = {})
+    allow(RepositoryForkWorker).to receive(:perform_async).and_return(true)
     Projects::ForkService.new(from_project, user, params).execute
   end
 end

spec/services/projects/update_mirror_service_spec.rb

+require 'spec_helper'
+
+describe Projects::UpdateMirrorService do
+  let(:project) { create(:project) }
+  let(:repository) { project.repository }
+  let(:mirror_user) { project.owner }
+  subject { described_class.new(project, mirror_user) }
+
+  before do
+    project.import_url = Project::UNKNOWN_IMPORT_URL
+    project.mirror = true
+    project.mirror_user = mirror_user
+    project.save
+  end
+
+  describe "#execute" do
+    it "fetches the upstream repository" do
+      expect(project).to receive(:fetch_mirror)
+
+      subject.execute
+    end
+
+    it "succeeds" do
+      allow(project).to receive(:fetch_mirror) { fetch_mirror(repository) }
+
+      result = subject.execute
+
+      expect(result[:status]).to eq(:success)
+    end
+
+    describe "updating tags" do
+      it "creates new tags" do
+        allow(project).to receive(:fetch_mirror) { fetch_mirror(repository) }
+
+        subject.execute
+
+        expect(repository.tag_names).to include('new-tag')
+      end
+    end
+
+    describe "updating branches" do
+      it "creates new branches" do
+        allow(project).to receive(:fetch_mirror) { fetch_mirror(repository) }
+
+        subject.execute
+
+        expect(repository.branch_names).to include('new-branch')
+      end
+
+      it "updates existing branches" do
+        allow(project).to receive(:fetch_mirror) { fetch_mirror(repository) }
+
+        subject.execute
+
+        expect(repository.find_branch('existing-branch').target).to eq(repository.find_branch('master').target)
+      end
+
+      it "doesn't update diverged branches" do
+        allow(project).to receive(:fetch_mirror) { fetch_mirror(repository) }
+
+        subject.execute
+
+        expect(repository.find_branch('markdown').target).not_to eq(repository.find_branch('master').target)
+      end
+    end
+
+    describe "when the mirror user doesn't have access" do
+      let(:mirror_user) { create(:user) }
+
+      it "fails" do
+        allow(project).to receive(:fetch_mirror) { fetch_mirror(repository) }
+
+        result = subject.execute
+
+        expect(result[:status]).to eq(:error)
+      end
+    end
+  end
+
+  def fetch_mirror(repository)
+    rugged = repository.rugged
+    masterrev = repository.find_branch('master').target
+
+    parentrev = repository.commit(masterrev).parent_id
+    rugged.references.create('refs/heads/existing-branch', parentrev)
+
+    repository.expire_branches_cache
+    repository.branches
+
+    # New branch
+    rugged.references.create('refs/remotes/upstream/new-branch', masterrev)
+
+    # Updated existing branch
+    rugged.references.create('refs/remotes/upstream/existing-branch', masterrev)
+
+    # Diverged branch
+    rugged.references.create('refs/remotes/upstream/markdown', masterrev)
+
+    # New tag
+    rugged.references.create('refs/tags/new-tag', masterrev)
+  end
+end

spec/workers/repository_fork_worker_spec.rb

@@ -12,7 +12,6 @@ describe RepositoryForkWorker do
                                                    project.path_with_namespace,
                                                    fork_project.namespace.path).
                                                    and_return(true)
-      expect(ProjectCacheWorker).to receive(:perform_async)
 
       subject.perform(project.id,
                       project.path_with_namespace,