Merge branch 'ci-secrets-docs' into 'master'

CI Secrets documentation

See merge request gitlab-org/gitlab!38552

doc/ci/examples/authenticating-with-hashicorp-vault/index.md

@@ -9,6 +9,12 @@ type: tutorial
 
 This tutorial demonstrates how to authenticate, configure, and read secrets with HashiCorp's Vault from GitLab CI/CD.
 
+NOTE: **Note:**
+[GitLab Premium](https://about.gitlab.com/pricing/) supports read access to a
+Hashicorp Vault, and enables you to
+[use Vault secrets in a CI job](../../secrets/index.md#use-vault-secrets-in-a-ci-job-premium).
+To learn more, read [Using external secrets in CI](../../secrets/index.md).
+
 ## Requirements
 
 This tutorial assumes you are familiar with GitLab CI/CD and Vault.

doc/ci/secrets/index.md

+---
+stage: Release
+group: Release Management
+info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers
+type: concepts, howto
+---
+
+# Using external secrets in CI
+
+> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/218746) in GitLab 13.4 and GitLab Runner 13.4.
+
+Secrets represent sensitive information your CI job needs to complete work. This
+sensitive information can be items like API tokens, database credentials, or private keys.
+Secrets are sourced from your secrets provider.
+
+Unlike CI variables, which are always presented to a job, secrets must be explicitly
+required by a job. Read [GitLab CI/CD pipeline configuration reference](../yaml/README.md#secrets)
+for more information about the syntax.
+
+GitLab has selected [Vault by Hashicorp](https://www.vaultproject.io) as the
+first supported provider, and [KV-V2](https://www.vaultproject.io/docs/secrets/kv/kv-v2)
+as the first supported secrets engine.
+
+GitLab authenticates using Vault's
+[JWT Auth method](https://www.vaultproject.io/docs/auth/jwt#jwt-authentication), using
+the [JSON Web Token](https://gitlab.com/gitlab-org/gitlab/-/issues/207125) (`CI_JOB_JWT`)
+introduced in GitLab 12.10.
+
+You must [configure your Vault server](#configure-your-vault-server) before you
+can use [use Vault secrets in a CI job](#use-vault-secrets-in-a-ci-job-premium).
+
+NOTE: **Note:**
+Read the [Authenticating and Reading Secrets With Hashicorp Vault](../examples/authenticating-with-hashicorp-vault/index.md)
+tutorial for a version of this feature that is available to all
+subscription levels, supports writing secrets to and deleting secrets from Vault,
+and multiple secrets engines.
+
+## Configure your Vault server
+
+To configure your Vault server:
+
+1. Enable the authentication method by running these commands. They provide your Vault
+   server the [JSON Web Key Set](https://tools.ietf.org/html/rfc7517) (JWKS) endpoint for your GitLab instance, so Vault
+   can fetch the public signing key and verify the JSON Web Token (JWT) when authenticating:
+
+   ```shell
+   $ vault auth enable jwt
+
+   $ vault write auth/jwt/config \
+     jwks_url="https://gitlab.example.com/-/jwks" \
+     bound_issuer="gitlab.example.com"
+   ```
+
+1. Configure policies on your Vault server to grant or forbid access to certain
+   paths and operations. This example grants read access to the set of secrets
+   required by your production environment:
+
+   ```shell
+   vault policy write myproject-production - <<EOF
+   # Read-only permission on 'ops/data/production/*' path
+
+   path "ops/data/production/*" {
+     capabilities = [ "read" ]
+   }
+   EOF
+   ```
+
+1. Configure roles on your Vault server, restricting roles to a project or namespace,
+   as described in [Configure Vault server roles](#configure-vault-server-roles) on this page.
+1. [Create the following CI variables](../variables/README.md#custom-environment-variables)
+   to provide details about your Vault server:
+   - `VAULT_SERVER_URL` - The URL of your Vault server, such as `https://vault.example.com:8200`.
+     Required.
+   - `VAULT_AUTH_ROLE` - (Optional) The role to use when attempting to authenticate.
+     If no role is specified, Vault uses the [default role](https://www.vaultproject.io/api/auth/jwt#default_role)
+     specified when the authentication method was configured.
+   - `VAULT_AUTH_PATH` - (Optional) The path where the authentication method is mounted, default is `jwt`.
+
+   NOTE: **Note:**
+   Support for [providing these values in the user interface](https://gitlab.com/gitlab-org/gitlab/-/issues/218677)
+   is planned but not yet implemented.
+
+## Use Vault secrets in a CI job **(PREMIUM)**
+
+> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/28321) in [GitLab Premium](https://about.gitlab.com/pricing/) 13.4 and GitLab Runner 13.4.
+
+After [configuring your Vault server](#configure-your-vault-server), you can use
+the secrets stored in Vault by defining them with the `vault` keyword:
+
+```yaml
+secrets:
+  DATABASE_PASSWORD:
+    vault: production/db/password@ops # translates to secret `ops/data/production/db`, field `password`
+```
+
+In this example:
+
+- `production/db` - The secret.
+- `password` The field.
+- `ops` - The path where the secrets engine is mounted.
+
+After GitLab fetches the secret from Vault, the value is saved in a temporary file.
+The path to this file is stored in environment variable named `DATABASE_PASSWORD`,
+similar to [CI variables of type `file`](../variables/README.md#custom-environment-variables-of-type-file).
+
+For more information about the supported syntax, read the
+[`.gitlab-ci.yml` reference](../yaml/README.md#secretsvault-premium).
+
+## Configure Vault server roles
+
+When a CI job attempts to authenticate, it specifies a role. You can use roles to group
+different policies together. If authentication is successful, these policies are
+attached to the resulting Vault token.
+
+[Bound claims](https://www.vaultproject.io/docs/auth/jwt#bound-claims) are predefined
+values that are matched to the JWT's claims. With bounded claims, you can restrict access
+to specific GitLab users, specific projects, or even jobs running for specific Git
+references. You can have as many bounded claims you need, but they must *all* match
+for authentication to be successful.
+
+Combining bounded claims with GitLab features like [user roles](../../user/permissions.md)
+and [protected branches](../../user/project/protected_branches.md), you can tailor
+these rules to fit your specific use case. In this example, authentication is allowed
+only for jobs running for protected tags with names matching the pattern used for
+production releases:
+
+```shell
+$ vault write auth/jwt/role/myproject-production - <<EOF
+{
+  "role_type": "jwt",
+  "policies": ["myproject-production"],
+  "token_explicit_max_ttl": 60,
+  "user_claim": "user_email",
+  "bound_claims_type": "glob",
+  "bound_claims": {
+    "project_id": "42",
+    "ref_protected": "true",
+    "ref_type": "tag",
+    "ref": "auto-deploy-*"
+  }
+}
+EOF
+```
+
+CAUTION: **Caution:**
+Always restrict your roles to a project or namespace by using one of the provided
+claims like `project_id` or `namespace_id`. Without these restrictions, any JWT
+generated by this GitLab instance may be allowed to authenticate using this role.
+
+For a full list of `CI_JOB_JWT` claims, read the
+[How it works](../examples/authenticating-with-hashicorp-vault/index.md#how-it-works) section of the
+[Authenticating and Reading Secrets With Hashicorp Vault](../examples/authenticating-with-hashicorp-vault/index.md) tutorial.
+
+You can also specify some attributes for the resulting Vault tokens, such as time-to-live,
+IP address range, and number of uses. The full list of options is available in
+[Vault's documentation on creating roles](https://www.vaultproject.io/api/auth/jwt#create-role)
+for the JSON web token method.

doc/ci/variables/README.md

@@ -431,7 +431,7 @@ script:
 You can define per-project or per-group variables
 that are set in the pipeline environment. Group-level variables are stored out of
 the repository (not in `.gitlab-ci.yml`) and are securely passed to GitLab Runner,
-which makes them available during a pipeline run. For Premium users who do **not** use an external key store or who use GitLab's [integration with HashiCorp Vault](../examples/authenticating-with-hashicorp-vault/index.md), we recommend using group environment variables to store secrets like passwords, SSH keys, and credentials.
+which makes them available during a pipeline run. For Premium users who do **not** use an external key store or who use GitLab's [integration with HashiCorp Vault](../secrets/index.md), we recommend using group environment variables to store secrets like passwords, SSH keys, and credentials.
 
 Group-level variables can be added by:
 

doc/ci/variables/predefined_variables.md

@@ -70,7 +70,7 @@ Kubernetes-specific environment variables are detailed in the
 | `CI_JOB_NAME`                                 | 9.0    | 0.5    | The name of the job as defined in `.gitlab-ci.yml`                                                                                                                                                                                                                                                                                                         |
 | `CI_JOB_STAGE`                                | 9.0    | 0.5    | The name of the stage as defined in `.gitlab-ci.yml`                                                                                                                                                                                                                                                                                                       |
 | `CI_JOB_TOKEN`                                | 9.0    | 1.2    | Token used for authenticating with the [GitLab Container Registry](../../user/packages/container_registry/index.md), downloading [dependent repositories](../../user/project/new_ci_build_permissions_model.md#dependent-repositories), and accessing [GitLab-managed Terraform state](../../user/infrastructure/index.md#gitlab-managed-terraform-state).         |
-| `CI_JOB_JWT`                                  | 12.10  | all    | RS256 JSON web token that can be used for authenticating with third party systems that support JWT authentication, for example [HashiCorp's Vault](../examples/authenticating-with-hashicorp-vault). |
+| `CI_JOB_JWT`                                  | 12.10  | all    | RS256 JSON web token that can be used for authenticating with third party systems that support JWT authentication, for example [HashiCorp's Vault](../secrets/index.md). |
 | `CI_JOB_URL`                                  | 11.1   | 0.5    | Job details URL                                                                                                                                                                                                                                                                                                                                            |
 | `CI_KUBERNETES_ACTIVE`                        | 13.0   | all    | Included with the value `true` only if the pipeline has a Kubernetes cluster available for deployments. Not included if no cluster is available. Can be used as an alternative to [`only:kubernetes`/`except:kubernetes`](../yaml/README.md#onlykubernetesexceptkubernetes) with [`rules:if`](../yaml/README.md#rulesif)                                    |
 | `CI_MERGE_REQUEST_ASSIGNEES`                  | 11.9   | all    | Comma-separated list of username(s) of assignee(s) for the merge request if [the pipelines are for merge requests](../merge_request_pipelines/index.md). Available only if `only: [merge_requests]` or [`rules`](../yaml/README.md#rules) syntax is used and the merge request is created.                                                                                                              |

doc/ci/yaml/README.md

@@ -4042,6 +4042,53 @@ The YAML described above would be translated into a CLI command like this:
 release-cli create --name "Release $CI_COMMIT_SHA" --description "Created using the release-cli $EXTRA_DESCRIPTION" --tag-name "v${MAJOR}.${MINOR}.${REVISION}" --ref "$CI_COMMIT_SHA" --released-at "2020-07-15T08:00:00Z" --milestone "m1" --milestone "m2" --milestone "m3"
 ```
 
+### `secrets`
+
+> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/33014) in GitLab 13.4.
+
+`secrets` indicates the [CI Secrets](../secrets/index.md) this job needs. It should be a hash,
+and the keys should be the names of the environment variables the job needs to access the secrets.
+
+#### `secrets:vault` **(PREMIUM)**
+
+> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/28321) in GitLab 13.4.
+
+`vault` keyword specifies secrets provided by [Hashicorp's Vault](https://www.vaultproject.io/).
+This syntax has multiple forms. The shortest form asssumes the use of the
+[KV-V2](https://www.vaultproject.io/docs/secrets/kv/kv-v2) secrets engine,
+mounted at the default path `kv-v2`. The last part of the secret's path is the
+field to fetch the value for:
+
+```yaml
+job:
+  secrets:
+    DATABASE_PASSWORD:
+      vault: production/db/password # translates to secret `kv-v2/data/production/db`, field `password`
+```
+
+You can specify a custom secrets engine path by adding a suffix starting with `@`:
+
+```yaml
+job:
+  secrets:
+    DATABASE_PASSWORD:
+      vault: production/db/password@ops # translates to secret `ops/data/production/db`, field `password`
+```
+
+In the detailed form of the syntax, you can specify all details explicitly:
+
+```yaml
+job:
+  secrets:
+    DATABASE_PASSWORD:      # translates to secret `ops/data/production/db`, field `password`
+      vault:
+        engine:
+          name: kv-v2
+          path: ops
+        path: production/db
+        field: password
+```
+
 ### `pages`
 
 `pages` is a special job that is used to upload static content to GitLab that

doc/user/packages/composer_repository/index.md

@@ -172,4 +172,4 @@ CAUTION: **Important:**
 Make sure to never commit the `auth.json` file to your repository. To install packages from a CI job,
 consider using the [`composer config`](https://getcomposer.org/doc/articles/handling-private-packages-with-satis.md#authentication) tool with your personal access token
 stored in a [GitLab CI/CD environment variable](../../../ci/variables/README.md) or in
-[Hashicorp Vault](../../../ci/examples/authenticating-with-hashicorp-vault/index.md).
+[Hashicorp Vault](../../../ci/secrets/index.md).