Merge branch '216785-use-ci-job-token-for-terraform-state-auth-2' into 'master'

Allow job token auth to terraform state API # 2

See merge request gitlab-org/gitlab!34618

changelogs/unreleased/216785-use-ci-job-token-for-terraform-state-auth.yml

+---
+title: Allow CI_JOB_TOKEN for authenticating to the Terraform state API
+merge_request: 34618
+author:
+type: added

doc/ci/variables/predefined_variables.md

@@ -64,7 +64,7 @@ You can add a command to your `.gitlab-ci.yml` file to
 | `CI_JOB_MANUAL`                               | 8.12   | all    | The flag to indicate that job was manually started                                                                                                                                                                                                                                                                                                         |
 | `CI_JOB_NAME`                                 | 9.0    | 0.5    | The name of the job as defined in `.gitlab-ci.yml`                                                                                                                                                                                                                                                                                                         |
 | `CI_JOB_STAGE`                                | 9.0    | 0.5    | The name of the stage as defined in `.gitlab-ci.yml`                                                                                                                                                                                                                                                                                                       |
-| `CI_JOB_TOKEN`                                | 9.0    | 1.2    | Token used for authenticating with the [GitLab Container Registry](../../user/packages/container_registry/index.md) and downloading [dependent repositories](../../user/project/new_ci_build_permissions_model.md#dependent-repositories)                                                                                                                                                                                                              |
+| `CI_JOB_TOKEN`                                | 9.0    | 1.2    | Token used for authenticating with the [GitLab Container Registry](../../user/packages/container_registry/index.md), downloading [dependent repositories](../../user/project/new_ci_build_permissions_model.md#dependent-repositories), and accessing [GitLab-managed Terraform state](../../user/infrastructure/index.md#gitlab-managed-terraform-state).         |
 | `CI_JOB_JWT`                                  | 12.10  | all    | RS256 JSON web token that can be used for authenticating with third party systems that support JWT authentication, for example [HashiCorp's Vault](../examples/authenticating-with-hashicorp-vault). |
 | `CI_JOB_URL`                                  | 11.1   | 0.5    | Job details URL                                                                                                                                                                                                                                                                                                                                            |
 | `CI_KUBERNETES_ACTIVE`                        | 13.0   | all    | Included with the value `true` only if the pipeline has a Kubernetes cluster available for deployments. Not included if no cluster is available. Can be used as an alternative to [`only:kubernetes`/`except:kubernetes`](../yaml/README.md#onlykubernetesexceptkubernetes) with [`rules:if`](../yaml/README.md#rulesif)                                    |

doc/user/infrastructure/index.md

@@ -25,7 +25,7 @@ Amazon S3 or Google Cloud Storage. Its features include:
 To get started with a GitLab-managed Terraform State, there are two different options:
 
 - [Use a local machine](#get-started-using-local-development).
-- [Use GitLab CI](#get-started-using-a-gitlab-ci).
+- [Use GitLab CI](#get-started-using-gitlab-ci).
 
 ## Get started using local development
 
@@ -44,10 +44,15 @@ local machine, this is a simple way to get started:
    }
    ```
 
+1. Create a [Personal Access Token](../profile/personal_access_tokens.md) with
+   the `api` scope. The Terraform backend is restricted to users with
+   [Maintainer access](../permissions.md) to the repository.
+
 1. On your local machine, run `terraform init`, passing in the following options,
-   replacing `<YOUR-PROJECT-NAME>` and `<YOUR-PROJECT-ID>` with the values for
-   your project. This command initializes your Terraform state, and stores that
-   state within your GitLab project. This example uses `gitlab.com`:
+   replacing `<YOUR-PROJECT-NAME>`, `<YOUR-PROJECT-ID>`,  `<YOUR-USERNAME>` and
+   `<YOUR-ACCESS-TOKEN>` with the relevant values. This command initializes your
+   Terraform state, and stores that state within your GitLab project. This example
+   uses `gitlab.com`:
 
    ```shell
    terraform init \
@@ -61,30 +66,24 @@ local machine, this is a simple way to get started:
        -backend-config="retry_wait_min=5"
    ```
 
-Next, [configure the backend](#configure-the-variables-and-backend).
+Next, [configure the backend](#configure-the-backend).
 
-## Get started using a GitLab CI
+## Get started using GitLab CI
 
 If you don't want to start with local development, you can also use GitLab CI to
 run your `terraform plan` and `terraform apply` commands.
 
-Next, [configure the backend](#configure-the-variables-and-backend).
+Next, [configure the backend](#configure-the-backend).
 
-## Configure the variables and backend
+## Configure the backend
 
-After executing the `terraform init` command, you must configure the needed CI
-variables, the Terraform backend, and the CI YAML file:
+After executing the `terraform init` command, you must configure the Terraform backend
+and the CI YAML file:
 
-1. Create a [Personal Access Token](../profile/personal_access_tokens.md) with
-   the `api` scope. The Terraform backend is restricted to tokens with
-   [Maintainer access](../permissions.md) to the repository.
-1. To keep the Personal Access Token secure, add it as a
-   [CI/CD environment variable](../../ci/variables/README.md). For the examples on
-   this page, it's set to the environment variable `GITLAB_TF_PASSWORD`.
+CAUTION: **Important:**
+The Terraform backend is restricted to users with [Maintainer access](../permissions.md)
+to the repository.
 
-   CAUTION: **Important:**
-   If you plan to use the environment variable on an unprotected branch, make sure
-   to set the variable protection settings correctly.
 1. In your Terraform project, define the [HTTP backend](https://www.terraform.io/docs/backends/types/http.html)
    by adding the following code block in a `.tf` file (such as `backend.tf`) to
    define the remote backend:
@@ -129,7 +128,7 @@ variables, the Terraform backend, and the CI YAML file:
    before_script:
      - cd ${TF_ROOT}
      - terraform --version
-     - terraform init -backend-config="address=${GITLAB_TF_ADDRESS}" -backend-config="lock_address=${GITLAB_TF_ADDRESS}/lock" -backend-config="unlock_address=${GITLAB_TF_ADDRESS}/lock" -backend-config="username=${GITLAB_USER_LOGIN}" -backend-config="password=${GITLAB_TF_PASSWORD}" -backend-config="lock_method=POST" -backend-config="unlock_method=DELETE" -backend-config="retry_wait_min=5"
+     - terraform init -backend-config="address=${GITLAB_TF_ADDRESS}" -backend-config="lock_address=${GITLAB_TF_ADDRESS}/lock" -backend-config="unlock_address=${GITLAB_TF_ADDRESS}/lock" -backend-config="username=gitlab-ci-token" -backend-config="password=${CI_JOB_TOKEN}" -backend-config="lock_method=POST" -backend-config="unlock_method=DELETE" -backend-config="retry_wait_min=5"
 
    stages:
      - validate

lib/api/terraform/state.rb

@@ -32,7 +32,7 @@ module API
           end
 
           desc 'Get a terraform state by its name'
-          route_setting :authentication, basic_auth_personal_access_token: true
+          route_setting :authentication, basic_auth_personal_access_token: true, job_token_allowed: :basic_auth
           get do
             remote_state_handler.find_with_lock do |state|
               no_content! unless state.file.exists?
@@ -44,7 +44,7 @@ module API
           end
 
           desc 'Add a new terraform state or update an existing one'
-          route_setting :authentication, basic_auth_personal_access_token: true
+          route_setting :authentication, basic_auth_personal_access_token: true, job_token_allowed: :basic_auth
           post do
             data = request.body.read
             no_content! if data.empty?
@@ -57,7 +57,7 @@ module API
           end
 
           desc 'Delete a terraform state of a certain name'
-          route_setting :authentication, basic_auth_personal_access_token: true
+          route_setting :authentication, basic_auth_personal_access_token: true, job_token_allowed: :basic_auth
           delete do
             remote_state_handler.handle_with_lock do |state|
               state.destroy!
@@ -66,7 +66,7 @@ module API
           end
 
           desc 'Lock a terraform state of a certain name'
-          route_setting :authentication, basic_auth_personal_access_token: true
+          route_setting :authentication, basic_auth_personal_access_token: true, job_token_allowed: :basic_auth
           params do
             requires :ID, type: String, limit: 255, desc: 'Terraform state lock ID'
             requires :Operation, type: String, desc: 'Terraform operation'
@@ -103,7 +103,7 @@ module API
           end
 
           desc 'Unlock a terraform state of a certain name'
-          route_setting :authentication, basic_auth_personal_access_token: true
+          route_setting :authentication, basic_auth_personal_access_token: true, job_token_allowed: :basic_auth
           params do
             optional :ID, type: String, limit: 255, desc: 'Terraform state lock ID'
           end

lib/gitlab/auth/auth_finders.rb

@@ -56,6 +56,7 @@ module Gitlab
 
       def find_user_from_job_token
         return unless route_authentication_setting[:job_token_allowed]
+        return find_user_from_basic_auth_job if route_authentication_setting[:job_token_allowed] == :basic_auth
 
         token = current_request.params[JOB_TOKEN_PARAM].presence ||
           current_request.params[RUNNER_JOB_TOKEN_PARAM].presence ||

spec/lib/gitlab/auth/auth_finders_spec.rb

@@ -4,6 +4,7 @@ require 'spec_helper'
 
 describe Gitlab::Auth::AuthFinders do
   include described_class
+  include HttpBasicAuthHelpers
 
   let(:user) { create(:user) }
   let(:env) do
@@ -22,10 +23,7 @@ describe Gitlab::Auth::AuthFinders do
   end
 
   def set_basic_auth_header(username, password)
-    set_header(
-      'HTTP_AUTHORIZATION',
-      ActionController::HttpAuthentication::Basic.encode_credentials(username, password)
-    )
+    env.merge!(basic_auth_header(username, password))
   end
 
   describe '#find_user_from_warden' do
@@ -653,6 +651,24 @@ describe Gitlab::Auth::AuthFinders do
       it_behaves_like 'job token params', described_class::JOB_TOKEN_PARAM
       it_behaves_like 'job token params', described_class::RUNNER_JOB_TOKEN_PARAM
     end
+
+    context 'when the job token is provided via basic auth' do
+      let(:route_authentication_setting) { { job_token_allowed: :basic_auth } }
+      let(:username) { Ci::Build::CI_REGISTRY_USER }
+      let(:token) { job.token }
+
+      before do
+        set_basic_auth_header(username, token)
+      end
+
+      it { is_expected.to eq(user) }
+
+      context 'credentials are provided but route setting is incorrect' do
+        let(:route_authentication_setting) { { job_token_allowed: :unknown } }
+
+        it { is_expected.to be_nil }
+      end
+    end
   end
 
   describe '#find_runner_from_token' do

spec/requests/api/oauth_tokens_spec.rb

@@ -3,20 +3,9 @@
 require 'spec_helper'
 
 describe 'OAuth tokens' do
-  context 'Resource Owner Password Credentials' do
-    def basic_auth_header(username, password)
-      {
-        'HTTP_AUTHORIZATION' => ActionController::HttpAuthentication::Basic.encode_credentials(
-          username,
-          password
-        )
-      }
-    end
-
-    def client_basic_auth_header(client)
-      basic_auth_header(client.uid, client.secret)
-    end
+  include HttpBasicAuthHelpers
 
+  context 'Resource Owner Password Credentials' do
     def request_oauth_token(user, headers = {})
       post '/oauth/token',
          params: { username: user.username, password: user.password, grant_type: 'password' },

spec/requests/api/terraform/state_spec.rb

@@ -3,6 +3,8 @@
 require 'spec_helper'
 
 describe API::Terraform::State do
+  include HttpBasicAuthHelpers
+
   let_it_be(:project) { create(:project) }
   let_it_be(:developer) { create(:user, developer_projects: [project]) }
   let_it_be(:maintainer) { create(:user, maintainer_projects: [project]) }
@@ -10,7 +12,7 @@ describe API::Terraform::State do
   let!(:state) { create(:terraform_state, :with_file, project: project) }
 
   let(:current_user) { maintainer }
-  let(:auth_header) { basic_auth_header(current_user) }
+  let(:auth_header) { user_basic_auth_header(current_user) }
   let(:project_id) { project.id }
   let(:state_name) { state.name }
   let(:state_path) { "/projects/#{project_id}/terraform/state/#{state_name}" }
@@ -23,7 +25,7 @@ describe API::Terraform::State do
     subject(:request) { get api(state_path), headers: auth_header }
 
     context 'without authentication' do
-      let(:auth_header) { basic_auth_header('failing_token') }
+      let(:auth_header) { basic_auth_header('bad', 'token') }
 
       it 'returns 401 if user is not authenticated' do
         request
@@ -32,34 +34,71 @@ describe API::Terraform::State do
       end
     end
 
-    context 'with maintainer permissions' do
-      let(:current_user) { maintainer }
+    context 'personal acceess token authentication' do
+      context 'with maintainer permissions' do
+        let(:current_user) { maintainer }
 
-      it 'returns terraform state belonging to a project of given state name' do
-        request
+        it 'returns terraform state belonging to a project of given state name' do
+          request
 
-        expect(response).to have_gitlab_http_status(:ok)
-        expect(response.body).to eq(state.file.read)
+          expect(response).to have_gitlab_http_status(:ok)
+          expect(response.body).to eq(state.file.read)
+        end
+
+        context 'for a project that does not exist' do
+          let(:project_id) { '0000' }
+
+          it 'returns not found' do
+            request
+
+            expect(response).to have_gitlab_http_status(:not_found)
+          end
+        end
       end
 
-      context 'for a project that does not exist' do
-        let(:project_id) { '0000' }
+      context 'with developer permissions' do
+        let(:current_user) { developer }
 
-        it 'returns not found' do
+        it 'returns forbidden if the user cannot access the state' do
           request
 
-          expect(response).to have_gitlab_http_status(:not_found)
+          expect(response).to have_gitlab_http_status(:forbidden)
         end
       end
     end
 
-    context 'with developer permissions' do
-      let(:current_user) { developer }
+    context 'job token authentication' do
+      let(:auth_header) { job_basic_auth_header(job) }
 
-      it 'returns forbidden if the user cannot access the state' do
-        request
+      context 'with maintainer permissions' do
+        let(:job) { create(:ci_build, project: project, user: maintainer) }
 
-        expect(response).to have_gitlab_http_status(:forbidden)
+        it 'returns terraform state belonging to a project of given state name' do
+          request
+
+          expect(response).to have_gitlab_http_status(:ok)
+          expect(response.body).to eq(state.file.read)
+        end
+
+        context 'for a project that does not exist' do
+          let(:project_id) { '0000' }
+
+          it 'returns not found' do
+            request
+
+            expect(response).to have_gitlab_http_status(:not_found)
+          end
+        end
+      end
+
+      context 'with developer permissions' do
+        let(:job) { create(:ci_build, project: project, user: developer) }
+
+        it 'returns forbidden if the user cannot access the state' do
+          request
+
+          expect(response).to have_gitlab_http_status(:forbidden)
+        end
       end
     end
   end

spec/support/helpers/api_helpers.rb

@@ -40,17 +40,6 @@ module ApiHelpers
     end
   end
 
-  def basic_auth_header(user = nil)
-    return { 'HTTP_AUTHORIZATION' => user } unless user.respond_to?(:username)
-
-    {
-      'HTTP_AUTHORIZATION' => ActionController::HttpAuthentication::Basic.encode_credentials(
-        user.username,
-        create(:personal_access_token, user: user).token
-      )
-    }
-  end
-
   def expect_empty_array_response
     expect_successful_response_with_paginated_array
     expect(json_response.length).to eq(0)

spec/support/helpers/http_basic_auth_helpers.rb

+# frozen_string_literal: true
+
+module HttpBasicAuthHelpers
+  def user_basic_auth_header(user)
+    access_token = create(:personal_access_token, user: user)
+
+    basic_auth_header(user.username, access_token.token)
+  end
+
+  def job_basic_auth_header(job)
+    basic_auth_header(Ci::Build::CI_REGISTRY_USER, job.token)
+  end
+
+  def client_basic_auth_header(client)
+    basic_auth_header(client.uid, client.secret)
+  end
+
+  def basic_auth_header(username, password)
+    {
+      'HTTP_AUTHORIZATION' => ActionController::HttpAuthentication::Basic.encode_credentials(
+        username,
+        password
+      )
+    }
+  end
+end