Squashed commit of the following:

commit e8ecae7e
Author: Shinya Maeda <shinya@gitlab.com>
Date:   Sat Jun 2 11:55:42 2018 +0900

    Reveert   build_relations and simply add a line for creating iid

commit b02b2602
Merge: c89e5784 fe0ebf76
Author: Shinya Maeda <shinya@gitlab.com>
Date:   Sat Jun 2 11:15:53 2018 +0900

    Merge branch 'master' into per-project-pipeline-iid

commit c89e5784
Author: Shinya Maeda <shinya@gitlab.com>
Date:   Fri Jun 1 15:46:15 2018 +0900

    Use shared examples for populate spec

commit c418d687
Author: Shinya Maeda <shinya@gitlab.com>
Date:   Fri Jun 1 15:41:33 2018 +0900

    Remove unneccesary spec

commit c754b693
Author: Shinya Maeda <shinya@gitlab.com>
Date:   Fri Jun 1 15:37:36 2018 +0900

    Clean up presence validation spec

commit f7f60ab5
Author: Shinya Maeda <shinya@gitlab.com>
Date:   Fri Jun 1 14:53:00 2018 +0900

    Add spec for variables expressions with pipeline iid

commit 4beeb602
Author: Shinya Maeda <shinya@gitlab.com>
Date:   Fri Jun 1 14:36:52 2018 +0900

    Fix populate_spec

commit 272b8dca
Merge: 0e22b50d 50fda506
Author: Shinya Maeda <shinya@gitlab.com>
Date:   Fri Jun 1 14:32:40 2018 +0900

    Merge branch 'master' into per-project-pipeline-iid

commit 0e22b50d
Author: Shinya Maeda <shinya@gitlab.com>
Date:   Wed May 30 16:42:55 2018 +0900

    Add spec for variables expression

commit 59e1e971
Author: Shinya Maeda <shinya@gitlab.com>
Date:   Wed May 30 15:17:09 2018 +0900

    Add build_relations method in  Chain::Populate

commit 09122f93
Merge: 1d20679e 5b1416aa
Author: Shinya Maeda <shinya@gitlab.com>
Date:   Wed May 30 14:50:09 2018 +0900

    Merge branch 'master' into per-project-pipeline-iid

commit 1d20679e
Merge: 8e92e25b 014f5f6a
Author: Shinya Maeda <shinya@gitlab.com>
Date:   Mon May 28 20:01:56 2018 +0900

    Merge branch 'master' into per-project-pipeline-iid

commit 8e92e25b
Author: Shinya Maeda <shinya@gitlab.com>
Date:   Tue May 22 15:21:45 2018 +0900

    Remvoe disable_ddl_transaction! and redandant RecordNotUnique exception rescue

commit f61666c0
Merge: 1c636b80 c6f72ac9
Author: Shinya Maeda <shinya@gitlab.com>
Date:   Tue May 22 14:32:40 2018 +0900

    Merge branch 'master' into per-project-pipeline-iid

commit 1c636b80
Merge: 82a49d0f 60b14e52
Author: Shinya Maeda <shinya@gitlab.com>
Date:   Thu May 17 15:51:33 2018 +0900

    Merge branch 'master' into per-project-pipeline-iid

commit 82a49d0f
Author: Shinya Maeda <shinya@gitlab.com>
Date:   Mon May 14 17:41:56 2018 +0900

    Clarify scope for AtomicInternalId shared spec

commit 1e76f289
Merge: a74184eb 40683268
Author: Shinya Maeda <shinya@gitlab.com>
Date:   Mon May 14 17:20:47 2018 +0900

    Merge branch 'master' into per-project-pipeline-iid

commit a74184eb
Author: Shinya Maeda <shinya@gitlab.com>
Date:   Fri May 11 16:52:48 2018 +0900

    Fix static analysys

commit 46fa3089
Author: Shinya Maeda <shinya@gitlab.com>
Date:   Fri May 11 16:49:18 2018 +0900

    Rescue RecordNotUnique when pipeline is created with non-unique iid

commit 910a7d02
Author: Shinya Maeda <shinya@gitlab.com>
Date:   Fri May 11 15:34:36 2018 +0900

    Remove numericality as it's redandant with integer column and validates nil IID

commit 9ccfcf55
Merge: 6a108b8f 35816eb7
Author: Shinya Maeda <shinya@gitlab.com>
Date:   Fri May 11 15:07:31 2018 +0900

    Merge branch 'master' into per-project-pipeline-iid

commit 6a108b8f
Author: Shinya Maeda <shinya@gitlab.com>
Date:   Thu May 10 16:22:43 2018 +0900

    Fix ensure_iid! method override problem

commit 30a6fb64
Author: Shinya Maeda <shinya@gitlab.com>
Date:   Thu May 10 15:56:47 2018 +0900

    Fix atomic internal id spec to allow generate pipeline

commit 40e17727
Merge: 7622f5ab 93498185
Author: Shinya Maeda <shinya@gitlab.com>
Date:   Thu May 10 14:30:45 2018 +0900

    Merge branch 'master' into per-project-pipeline-iid

commit 7622f5ab
Merge: 04dc80db 5b0e96d0
Author: Shinya Maeda <shinya@gitlab.com>
Date:   Wed May 9 14:46:41 2018 +0900

    Merge branch 'master' into per-project-pipeline-iid

commit 04dc80db
Author: Shinya Maeda <shinya@gitlab.com>
Date:   Tue May 8 16:01:18 2018 +0900

    Fix spec

commit 0af2ab18
Author: Shinya Maeda <shinya@gitlab.com>
Date:   Tue May 8 14:57:41 2018 +0900

    Decouple to_params from AtomicInternalId concern

commit 632b87a8
Merge: 07d1d8bd 0eb74426
Author: Shinya Maeda <shinya@gitlab.com>
Date:   Tue May 8 14:32:45 2018 +0900

    Merge branch 'master' into per-project-pipeline-iid

commit 07d1d8bd
Author: Kamil Trzciński <ayufan@ayufan.eu>
Date:   Thu May 3 10:50:57 2018 +0200

    Use **CI_PIPELINE_IID**

commit 58f229af
Author: Kamil Trzciński <ayufan@ayufan.eu>
Date:   Thu May 3 10:48:23 2018 +0200

    Make Atomic Internal ID work for pipelines

commit 35cf604b
Author: Shinya Maeda <shinya@gitlab.com>
Date:   Mon Apr 30 22:25:56 2018 +0900

    Add to_param override to lookup :id path

commit 787eddd5
Author: Shinya Maeda <shinya@gitlab.com>
Date:   Wed Apr 25 22:00:11 2018 +0900

    Revert column name change

commit 8327ad8d
Author: Shinya Maeda <shinya@gitlab.com>
Date:   Wed Apr 25 21:44:05 2018 +0900

    Add changelog

commit 8192cd70
Author: Shinya Maeda <shinya@gitlab.com>
Date:   Wed Apr 25 21:42:46 2018 +0900

    Expose CI_PIPELINE_IID_PER_PROJECT variable

commit 332d3d0e
Author: Shinya Maeda <shinya@gitlab.com>
Date:   Wed Apr 25 21:03:35 2018 +0900

    Change column name to iid_per_project. Add index to project_id and iid

commit 3248ca04
Author: Shinya Maeda <shinya@gitlab.com>
Date:   Tue Apr 24 17:08:43 2018 +0900

    Add per-project pipeline id

app/models/ci/pipeline.rb

@@ -7,6 +7,7 @@ module Ci
     include Presentable
     include Gitlab::OptimisticLocking
     include Gitlab::Utils::StrongMemoize
+    include AtomicInternalId
 
     prepend ::EE::Ci::Pipeline
 
@@ -15,6 +16,10 @@ module Ci
     belongs_to :auto_canceled_by, class_name: 'Ci::Pipeline'
     belongs_to :pipeline_schedule, class_name: 'Ci::PipelineSchedule'
 
+    has_internal_id :iid, scope: :project, presence: false, init: ->(s) do
+      s&.project&.pipelines&.maximum(:iid) || s&.project&.pipelines&.count
+    end
+
     has_one :source_pipeline, class_name: Ci::Sources::Pipeline
 
     has_many :sourced_pipelines, class_name: Ci::Sources::Pipeline, foreign_key: :source_pipeline_id
@@ -549,6 +554,7 @@ module Ci
 
     def predefined_variables
       Gitlab::Ci::Variables::Collection.new
+        .append(key: 'CI_PIPELINE_IID', value: iid.to_s)
         .append(key: 'CI_CONFIG_PATH', value: ci_yaml_file_path)
         .append(key: 'CI_PIPELINE_SOURCE', value: source.to_s)
         .append(key: 'CI_COMMIT_MESSAGE', value: git_commit_message)

app/models/concerns/atomic_internal_id.rb

@@ -25,9 +25,13 @@ module AtomicInternalId
   extend ActiveSupport::Concern
 
   module ClassMethods
-    def has_internal_id(column, scope:, init:) # rubocop:disable Naming/PredicateName
-      before_validation(on: :create) do
+    def has_internal_id(column, scope:, init:, presence: true) # rubocop:disable Naming/PredicateName
+      before_validation :"ensure_#{scope}_#{column}!", on: :create
+      validates column, presence: presence
+
+      define_method("ensure_#{scope}_#{column}!") do
         scope_value = association(scope).reader
+
         if read_attribute(column).blank? && scope_value
           scope_attrs = { scope_value.class.table_name.singularize.to_sym => scope_value }
           usage = self.class.table_name.to_sym
@@ -35,13 +39,9 @@ module AtomicInternalId
           new_iid = InternalId.generate_next(self, scope_attrs, usage, init)
           write_attribute(column, new_iid)
         end
-      end
 
-      validates column, presence: true, numericality: true
+        read_attribute(column)
+      end
     end
   end
-
-  def to_param
-    iid.to_s
-  end
 end

app/models/concerns/iid_routes.rb

+module IidRoutes
+  ##
+  # This automagically enforces all related routes to use `iid` instead of `id`
+  # If you want to use `iid` for some routes and `id` for other routes, this module should not to be included,
+  # instead you should define `iid` or `id` explictly at each route generators. e.g. pipeline_path(project.id, pipeline.iid)
+  def to_param
+    iid.to_s
+  end
+end

app/models/deployment.rb

 class Deployment < ActiveRecord::Base
   include AtomicInternalId
+  include IidRoutes
 
   belongs_to :project, required: true
   belongs_to :environment, required: true

app/models/internal_id.rb

@@ -14,7 +14,7 @@ class InternalId < ActiveRecord::Base
   belongs_to :project
   belongs_to :namespace
 
-  enum usage: { issues: 0, merge_requests: 1, deployments: 2, milestones: 3, epics: 4 }
+  enum usage: { issues: 0, merge_requests: 1, deployments: 2, milestones: 3, epics: 4, ci_pipelines: 5 }
 
   validates :usage, presence: true
 

app/models/issue.rb

@@ -5,6 +5,7 @@ class Issue < ActiveRecord::Base
   prepend EE::RelativePositioning
 
   include AtomicInternalId
+  include IidRoutes
   include Issuable
   include Noteable
   include Referable

app/models/merge_request.rb

 class MergeRequest < ActiveRecord::Base
   include AtomicInternalId
+  include IidRoutes
   include Issuable
   include Noteable
   include Referable

app/models/milestone.rb

@@ -9,6 +9,7 @@ class Milestone < ActiveRecord::Base
 
   include CacheMarkdownField
   include AtomicInternalId
+  include IidRoutes
   include Sortable
   include Referable
   include StripAttribute

changelogs/unreleased/per-project-pipeline-iid.yml

+---
+title: Add per-project pipeline id
+merge_request: 18558
+author:
+type: added

db/migrate/20180424160449_add_pipeline_iid_to_ci_pipelines.rb

+class AddPipelineIidToCiPipelines < ActiveRecord::Migration
+  include Gitlab::Database::MigrationHelpers
+
+  DOWNTIME = false
+
+  def up
+    add_column :ci_pipelines, :iid, :integer
+  end
+
+  def down
+    remove_column :ci_pipelines, :iid, :integer
+  end
+end

db/migrate/20180425205249_add_index_constraints_to_pipeline_iid.rb

+class AddIndexConstraintsToPipelineIid < ActiveRecord::Migration
+  include Gitlab::Database::MigrationHelpers
+
+  DOWNTIME = false
+
+  disable_ddl_transaction!
+
+  def up
+    add_concurrent_index :ci_pipelines, [:project_id, :iid], unique: true, where: 'iid IS NOT NULL'
+  end
+
+  def down
+    remove_concurrent_index :ci_pipelines, [:project_id, :iid]
+  end
+end

db/schema.rb

@@ -547,10 +547,12 @@ ActiveRecord::Schema.define(version: 20180529093006) do
     t.integer "config_source"
     t.boolean "protected"
     t.integer "failure_reason"
+    t.integer "iid"
   end
 
   add_index "ci_pipelines", ["auto_canceled_by_id"], name: "index_ci_pipelines_on_auto_canceled_by_id", using: :btree
   add_index "ci_pipelines", ["pipeline_schedule_id"], name: "index_ci_pipelines_on_pipeline_schedule_id", using: :btree
+  add_index "ci_pipelines", ["project_id", "iid"], name: "index_ci_pipelines_on_project_id_and_iid", unique: true, where: "(iid IS NOT NULL)", using: :btree
   add_index "ci_pipelines", ["project_id", "ref", "status", "id"], name: "index_ci_pipelines_on_project_id_and_ref_and_status_and_id", using: :btree
   add_index "ci_pipelines", ["project_id", "sha"], name: "index_ci_pipelines_on_project_id_and_sha", using: :btree
   add_index "ci_pipelines", ["project_id"], name: "index_ci_pipelines_on_project_id", using: :btree

doc/ci/variables/README.md

@@ -74,6 +74,7 @@ future GitLab releases.**
 | **CI_RUNNER_REVISION**          | all    | 10.6   | GitLab Runner revision that is executing the current job |
 | **CI_RUNNER_EXECUTABLE_ARCH**   | all    | 10.6   | The OS/architecture of the GitLab Runner executable (note that this is not necessarily the same as the environment of the executor) |
 | **CI_PIPELINE_ID**              | 8.10   | 0.5    | The unique id of the current pipeline that GitLab CI uses internally |
+| **CI_PIPELINE_IID**             | 11.0   | all    | The unique id of the current pipeline scoped to project |
 | **CI_PIPELINE_TRIGGERED**       | all    | all    | The flag to indicate that job was [triggered] |
 | **CI_PIPELINE_SOURCE**          | 10.0   | all    | Indicates how the pipeline was triggered. Possible options are: `push`, `web`, `trigger`, `schedule`, `api`, and `pipeline`. For pipelines created before GitLab 9.5, this will show as `unknown` |
 | **CI_PROJECT_DIR**              | all    | all    | The full path where the repository is cloned and where the job is run |

lib/gitlab/ci/pipeline/chain/populate.rb

@@ -8,6 +8,9 @@ module Gitlab
           PopulateError = Class.new(StandardError)
 
           def perform!
+            # Allocate next IID. This operation must be outside of transactions of pipeline creations.
+            pipeline.ensure_project_iid!
+
             ##
             # Populate pipeline with block argument of CreatePipelineService#execute.
             #

spec/lib/gitlab/ci/pipeline/chain/populate_spec.rb

@@ -42,6 +42,10 @@ describe Gitlab::Ci::Pipeline::Chain::Populate do
     it 'correctly assigns user' do
       expect(pipeline.builds).to all(have_attributes(user: user))
     end
+
+    it 'has pipeline iid' do
+      expect(pipeline.iid).to be > 0
+    end
   end
 
   context 'when pipeline is empty' do
@@ -68,6 +72,10 @@ describe Gitlab::Ci::Pipeline::Chain::Populate do
       expect(pipeline.errors.to_a)
         .to include 'No stages / jobs for this pipeline.'
     end
+
+    it 'wastes pipeline iid' do
+      expect(InternalId.ci_pipelines.where(project_id: project.id).last.last_value).to be > 0
+    end
   end
 
   context 'when pipeline has validation errors' do
@@ -87,6 +95,10 @@ describe Gitlab::Ci::Pipeline::Chain::Populate do
       expect(pipeline.errors.to_a)
         .to include 'Failed to build the pipeline!'
     end
+
+    it 'wastes pipeline iid' do
+      expect(InternalId.ci_pipelines.where(project_id: project.id).last.last_value).to be > 0
+    end
   end
 
   context 'when there is a seed blocks present' do
@@ -111,6 +123,12 @@ describe Gitlab::Ci::Pipeline::Chain::Populate do
         expect(pipeline.variables.first.key).to eq 'VAR'
         expect(pipeline.variables.first.value).to eq '123'
       end
+
+      it 'has pipeline iid' do
+        step.perform!
+
+        expect(pipeline.iid).to be > 0
+      end
     end
 
     context 'when seeds block tries to persist some resources' do
@@ -121,6 +139,12 @@ describe Gitlab::Ci::Pipeline::Chain::Populate do
       it 'raises exception' do
         expect { step.perform! }.to raise_error(ActiveRecord::RecordNotSaved)
       end
+
+      it 'wastes pipeline iid' do
+        expect { step.perform! }.to raise_error
+
+        expect(InternalId.ci_pipelines.where(project_id: project.id).exists?).to be_truthy
+      end
     end
   end
 
@@ -132,22 +156,39 @@ describe Gitlab::Ci::Pipeline::Chain::Populate do
     end
   end
 
-  context 'when using only/except build policies' do
-    let(:config) do
-      { rspec: { script: 'rspec', stage: 'test', only: ['master'] },
-        prod: { script: 'cap prod', stage: 'deploy', only: ['tags'] } }
-    end
+  context 'when variables policy is specified' do
+    shared_examples_for 'populates pipeline according to used policies' do
+      it 'populates pipeline according to used policies' do
+        step.perform!
 
-    let(:pipeline) do
-      build(:ci_pipeline, ref: 'master', project: project, config: config)
+        expect(pipeline.stages.size).to eq 1
+        expect(pipeline.stages.first.builds.size).to eq 1
+        expect(pipeline.stages.first.builds.first.name).to eq 'rspec'
+      end
     end
 
-    it 'populates pipeline according to used policies' do
-      step.perform!
+    context 'when using only/except build policies' do
+      let(:config) do
+        { rspec: { script: 'rspec', stage: 'test', only: ['master'] },
+          prod: { script: 'cap prod', stage: 'deploy', only: ['tags'] } }
+      end
+
+      let(:pipeline) do
+        build(:ci_pipeline, ref: 'master', project: project, sconfig: config)
+      end
 
-      expect(pipeline.stages.size).to eq 1
-      expect(pipeline.stages.first.builds.size).to eq 1
-      expect(pipeline.stages.first.builds.first.name).to eq 'rspec'
+      it_behaves_like 'populates pipeline according to used policies'
+
+      context 'when variables expression is specified' do
+        context 'when pipeline iid is the subject' do
+          let(:config) do
+            { rspec: { script: 'rspec', only: { variables: ["$CI_PIPELINE_IID == '1'"] } },
+              prod: { script: 'cap prod', only: { variables: ["$CI_PIPELINE_IID == '1000'"] } } }
+          end
+
+          it_behaves_like 'populates pipeline according to used policies'
+        end
+      end
     end
   end
 end

spec/lib/gitlab/import_export/safe_model_attributes.yml

@@ -245,6 +245,7 @@ Ci::Pipeline:
 - config_source
 - failure_reason
 - protected
+- iid
 Ci::Stage:
 - id
 - name

spec/models/ci/build_spec.rb

@@ -1579,6 +1579,7 @@ describe Ci::Build do
         { key: 'CI_PROJECT_NAMESPACE', value: project.namespace.full_path, public: true },
         { key: 'CI_PROJECT_URL', value: project.web_url, public: true },
         { key: 'CI_PROJECT_VISIBILITY', value: 'private', public: true },
+        { key: 'CI_PIPELINE_IID', value: pipeline.iid.to_s, public: true },
         { key: 'CI_CONFIG_PATH', value: pipeline.ci_yaml_file_path, public: true },
         { key: 'CI_PIPELINE_SOURCE', value: pipeline.source, public: true },
         { key: 'CI_COMMIT_MESSAGE', value: pipeline.git_commit_message, public: true },

spec/models/ci/pipeline_spec.rb

@@ -39,6 +39,16 @@ describe Ci::Pipeline, :mailer do
     end
   end
 
+  describe 'modules' do
+    it_behaves_like 'AtomicInternalId', validate_presence: false do
+      let(:internal_id_attribute) { :iid }
+      let(:instance) { build(:ci_pipeline) }
+      let(:scope) { :project }
+      let(:scope_attrs) { { project: instance.project } }
+      let(:usage) { :ci_pipelines }
+    end
+  end
+
   describe '#source' do
     context 'when creating new pipeline' do
       let(:pipeline) do
@@ -199,7 +209,8 @@ describe Ci::Pipeline, :mailer do
     it 'includes all predefined variables in a valid order' do
       keys = subject.map { |variable| variable[:key] }
 
-      expect(keys).to eq %w[CI_CONFIG_PATH
+      expect(keys).to eq %w[CI_PIPELINE_IID
+                            CI_CONFIG_PATH
                             CI_PIPELINE_SOURCE
                             CI_COMMIT_MESSAGE
                             CI_COMMIT_TITLE

spec/models/deployment_spec.rb

@@ -20,6 +20,7 @@ describe Deployment do
     it_behaves_like 'AtomicInternalId' do
       let(:internal_id_attribute) { :iid }
       let(:instance) { build(:deployment) }
+      let(:scope) { :project }
       let(:scope_attrs) { { project: instance.project } }
       let(:usage) { :deployments }
     end

spec/models/issue_spec.rb

@@ -17,6 +17,7 @@ describe Issue do
     it_behaves_like 'AtomicInternalId' do
       let(:internal_id_attribute) { :iid }
       let(:instance) { build(:issue) }
+      let(:scope) { :project }
       let(:scope_attrs) { { project: instance.project } }
       let(:usage) { :issues }
     end

spec/models/merge_request_spec.rb

@@ -84,6 +84,7 @@ describe MergeRequest do
     it_behaves_like 'AtomicInternalId' do
       let(:internal_id_attribute) { :iid }
       let(:instance) { build(:merge_request) }
+      let(:scope) { :target_project }
       let(:scope_attrs) { { project: instance.target_project } }
       let(:usage) { :merge_requests }
     end

spec/models/milestone_spec.rb

@@ -6,6 +6,7 @@ describe Milestone do
       it_behaves_like 'AtomicInternalId' do
         let(:internal_id_attribute) { :iid }
         let(:instance) { build(:milestone, project: build(:project), group: nil) }
+        let(:scope) { :project }
         let(:scope_attrs) { { project: instance.project } }
         let(:usage) { :milestones }
       end
@@ -15,6 +16,7 @@ describe Milestone do
       it_behaves_like 'AtomicInternalId' do
         let(:internal_id_attribute) { :iid }
         let(:instance) { build(:milestone, project: nil, group: build(:group)) }
+        let(:scope) { :group }
         let(:scope_attrs) { { namespace: instance.group } }
         let(:usage) { :milestones }
       end

spec/support/shared_examples/models/atomic_internal_id_spec.rb

 require 'spec_helper'
 
-shared_examples_for 'AtomicInternalId' do
+shared_examples_for 'AtomicInternalId' do |validate_presence: true|
   describe '.has_internal_id' do
     describe 'Module inclusion' do
       subject { described_class }
@@ -9,14 +9,31 @@ shared_examples_for 'AtomicInternalId' do
     end
 
     describe 'Validation' do
-      subject { instance }
-
       before do
-        allow(InternalId).to receive(:generate_next).and_return(nil)
+        allow_any_instance_of(described_class).to receive(:"ensure_#{scope}_#{internal_id_attribute}!")
+
+        instance.valid?
       end
 
-      it { is_expected.to validate_presence_of(internal_id_attribute) }
-      it { is_expected.to validate_numericality_of(internal_id_attribute) }
+      context 'when presence validation is required' do
+        before do
+          skip unless validate_presence
+        end
+
+        it 'validates presence' do
+          expect(instance.errors[internal_id_attribute]).to include("can't be blank")
+        end
+      end
+
+      context 'when presence validation is not required' do
+        before do
+          skip if validate_presence
+        end
+
+        it 'does not validate presence' do
+          expect(instance.errors[internal_id_attribute]).to be_empty
+        end
+      end
     end
 
     describe 'Creating an instance' do