Merge branch 'jcunha/update-vendored-cluster-management-proj-tpl-docs' into 'master'

Document the new Cluster Management Project Template

See merge request gitlab-org/gitlab!63660

doc/user/clusters/applications.md

@@ -6,6 +6,10 @@ info: To determine the technical writer assigned to the Stage/Group associated w
 
 # GitLab Managed Apps (DEPRECATED) **(FREE)**
 
+NOTE:
+The new recommended way to manage cluster applications is to use the [cluster management project template](management_project_template.md).
+If you want to migrate your GitLab managed apps management to this template, reference to [migrating from GitLab managed apps to project template](migrating_from_gma_to_project_template.md).
+
 **GitLab Managed Apps** was created to help you configure applications in your
 cluster directly from GitLab. You could use this feature through two different
 methods: "one-click install" and "CI/CD template". Both methods are **deprecated**:
@@ -25,9 +29,6 @@ learn how to proceed to keep your apps up and running:
 - [One-click install method](#install-with-one-click-deprecated)
 - [CI/CD template method](#install-using-gitlab-cicd-deprecated)
 
-NOTE:
-Despite being deprecated, the recommended way for installing GitLab integrated applications is by the GitLab CI/CD method presented below. We are working on a [cluster management project template](https://gitlab.com/gitlab-org/gitlab/-/issues/327908) with a simple upgrade path from the CI/CD based method.
-
 ## Install using GitLab CI/CD (DEPRECATED)
 
 > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/20822) in GitLab 12.6.

doc/user/clusters/integrations.md

@@ -14,6 +14,17 @@ To enable cluster integrations, first add a Kubernetes cluster to a GitLab
 [group](../group/clusters/index.md#group-level-kubernetes-clusters) or
 [instance](../instance/clusters/index.md).
 
+You can install your applications manually as shown in the following sections, or use the
+[Cluster management project template](management_project_template.md) that automates the
+installation.
+
+Although, the [Cluster management project template](management_project_template.md) still
+requires that you manually do the last steps of these sections,
+[Enable Prometheus integration for your cluster](#enable-prometheus-integration-for-your-cluster)
+or [Enable Elastic Stack integration for your cluster](#enable-elastic-stack-integration-for-your-cluster)
+depending on which application you are installing. We plan to also automate this step in the future,
+see the [opened issue](https://gitlab.com/gitlab-org/gitlab/-/issues/326565).
+
 ## Prometheus cluster integration
 
 > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/55244) in GitLab 13.11.

doc/user/clusters/management_project.md

@@ -16,7 +16,7 @@ privileges.
 
 This can be useful for:
 
-- Creating pipelines to install cluster-wide applications into your cluster, see [Install using GitLab CI/CD (beta)](applications.md#install-using-gitlab-cicd-deprecated) for details.
+- Creating pipelines to install cluster-wide applications into your cluster, see [management project template](management_project_template.md) for details.
 - Any jobs that require `cluster-admin` privileges.
 
 ## Permissions

doc/user/clusters/management_project_template.md

+---
+stage: Configure
+group: Configure
+info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments
+---
+
+# Cluster Management Project Template **(FREE)**
+
+> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/25318) in GitLab 12.10 with Helmfile support via Helm v2.
+> - [Improved](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/63577) in GitLab 14.0 with Helmfile support via Helm v3 instead, and a much more flexible usage of Helmfile. This introduces breaking changes that are detailed below.
+
+This [GitLab built-in project template](../project/working_with_projects.md#built-in-templates)
+provides a quicker start for users interested in managing cluster
+applications via [Helm v3](https://helm.sh/) charts. More specifically, taking advantage of the
+[Helmfile](https://github.com/roboll/helmfile) utility client. The template consists of some pre-configured apps that
+should help you get started quickly using various GitLab features. Still, you have all the flexibility to remove the ones you do not
+need, or even add new ones that are not built-in.
+
+## How to use this template
+
+1. Create a new project, choosing "GitLab Cluster Management" from the list of [built-in project templates](../project/working_with_projects.md#built-in-templates). 
+1. Make this project a [cluster management project](management_project.md).
+1. If you used the [GitLab Managed Apps](applications.md), refer to
+   [Migrating from GitLab Manged Apps](migrating_from_gma_to_project_template.md).
+
+### Components
+
+In the repository of the newly-created project, you will find:
+
+- A predefined [`.gitlab-ci.yml`](https://gitlab.com/gitlab-org/project-templates/cluster-management/-/blob/master/.gitlab-ci.yml)
+  file, with a CI pipeline already configured.
+- A main [`helmfile.yaml`](https://gitlab.com/gitlab-org/project-templates/cluster-management/-/blob/master/helmfile.yaml) to toggle which applications you would like to manage.
+- An `applications` directory with a `helmfile.yaml` configured for each application GitLab provides.
+
+#### The `.gitlab-ci.yml` file
+
+The base image used in your pipeline is built by the [cluster-applications](https://gitlab.com/gitlab-org/cluster-integration/cluster-applications)
+project. This image consists of a set of Bash utility scripts to support [Helm v3 releases](https://helm.sh/docs/intro/using_helm/#three-big-concepts):
+
+- `gl-fail-if-helm2-releases-exist {namespace}`: It tries to detect whether you have apps deployed through Helm v2 
+  releases for a given namespace. If so, it will fail the pipeline and ask you to manually 
+  [migrate your Helm v2 releases to Helm v3](https://helm.sh/docs/topics/v2_v3_migration/).
+- `gl-ensure-namespace {namespace}`: It creates the given namespace if it does not exist and adds the necessary label 
+  for the [Cilium](https://github.com/cilium/cilium/) app network policies to work.
+- `gl-adopt-resource-with-helm-v3 {arguments}`: Used only internally in the [cert-manager's](https://cert-manager.io/) Helmfile to
+  facilitate the GitLab Managed Apps adoption.
+- `gl-adopt-crds-with-helm-v3 {arguments}`: Used only internally in the [cert-manager's](https://cert-manager.io/) Helmfile to
+  facilitate the GitLab Managed Apps adoption.
+- `gl-helmfile {arguments}`: A thin wrapper that triggers the [Helmfile](https://github.com/roboll/helmfile) command.
+
+#### The main `helmfile.yml` file
+
+This file has a list of paths to other Helmfiles for each app. They're all commented out by default, so you must uncomment 
+the paths for the apps that you would like to manage.
+
+By default, each `helmfile.yaml` in these sub-paths will have the attribute `installed: true`, which signifies that everytime
+the pipeline runs, Helmfile will try to either install or update your apps according to the current state of your
+cluster and Helm releases. If you change this attribute to `installed: false`, Helmfile will try to uninstall this app
+from your cluster. [Read more](https://github.com/roboll/helmfile) about how Helmfile works.
+
+Furthermore, each app has an `applications/{app}/values.yaml` file. This is the 
+place where you can define some default values for your app's Helm chart. Some apps will already have defaults
+pre-defined by GitLab.
+
+#### Built-in applications
+
+The built-in applications are intended to provide an easy way to get started with various Kubernetes oriented GitLab features.
+
+The [built-in supported applications](https://gitlab.com/gitlab-org/project-templates/cluster-management/-/tree/master/applications) are:
+
+- Apparmor
+- Cert-manager
+- Cilium
+- Elastic Stack
+- Falco
+- Fluentd
+- GitLab Runner
+- Ingress
+- Prometheus
+- Sentry
+- Vault
+
+### Migrating from GitLab Managed Apps
+
+If you had GitLab Managed Apps, either One-Click or CI/CD install, read the docs on how to 
+[migrate from GitLab Managed Apps to project template](migrating_from_gma_to_project_template.md)

doc/user/clusters/migrating_from_gma_to_project_template.md

+---
+stage: Configure
+group: Configure
+info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments
+---
+
+# Migrating from GitLab Managed Apps to a management project template
+
+The [GitLab Managed Apps](applications.md) are deprecated in GitLab 14.0. To migrate to the new way of managing them:
+
+1. Read how the [management project template](management_project_template.md) works, and
+   create a new project based on the "GitLab Cluster Management" template.
+1. Create a new project as explained in the [management project template](management_project_template.md).
+1. Detect apps deployed through Helm v2 releases by using the pre-configured [`.gitlab-ci.yml`](management_project_template.md#the-gitlab-ciyml-file) file:
+    - In case you had overwritten the default GitLab Managed Apps namespace, edit `.gitlab-ci.yml`,
+      and make sure the script is receiving the correct namespace as an argument:
+
+      ```yaml
+      script:
+        - gl-fail-if-helm2-releases-exist <your_custom_namespace>
+      ```
+
+    - If you kept the default name (`gitlab-managed-apps`), then the script is already
+      set up.
+
+   Either way, [run a pipeline manually](../../ci/pipelines/index.md#run-a-pipeline-manually) and read the logs of the
+   `detect-helm2-releases` job to know if you have any Helm v2 releases and which are they.
+
+1. If you have no Helm v2 releases, skip this step. Otherwise, follow the official Helm docs on
+   [how to migrate from Helm v2 to Helm v3](https://helm.sh/blog/migrate-from-helm-v2-to-helm-v3/),
+   and clean up the Helm v2 releases after you are confident that they have been successfully migrated.
+
+1. In this step you should already have only Helm v3 releases.
+   Uncomment from the main [`./helmfile.yaml`](management_project_template.md#the-main-helmfileyml-file) the paths for the
+   applications that you would like to manage with this project. Although you could uncomment all the ones you want to
+   managed at once, we recommend you repeat the following steps separately for each app, so you do not get lost during
+   the process.
+1. Edit the associated `applications/{app}/helmfiles.yaml` to match the chart version currently deployed 
+   for your app. Take a GitLab Runner Helm v3 release as an example:
+   
+   The following command lists the releases and their versions:
+
+   ```shell
+   helm ls -n gitlab-managed-apps
+   
+   NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
+   runner gitlab-managed-apps 1 2021-06-09 19:36:55.739141644 +0000 UTC deployed gitlab-runner-0.28.0 13.11.0
+   ```
+
+   Take the version from the `CHART` column which is in the format `{release}-v{chart_version}`,
+   then edit the `version:` attribute in the `./applications/gitlab-runner/helmfile.yaml`, so that it matches the version
+   you have currently deployed. This is a safe step to avoid upgrading versions during this migration.
+   Make sure you replace `gitlab-managed-apps` from the above command if you have your apps deployed to a different
+   namespace.
+
+1. Edit the `applications/{app}/values.yaml` associated with your app to match the currently
+   deployed values. For example, for GitLab Runner:
+
+    1. Copy the output of the following command (it might be big):
+
+   ```shell
+   helm get values runner -n gitlab-managed-apps -a --output yaml
+   ```
+
+    1. Overwrite `applications/gitlab-runner/values.yaml` with the output of the previous command.
+
+   This safe step will guarantee that no unexpected default values overwrite your currently deployed values.
+   For instance, your GitLab Runner could have its `gitlabUrl` or `runnerRegistrationToken` overwritten by mistake.
+
+1. Some apps require special attention:
+
+    - Ingress: Due to an existing [chart issue](https://github.com/helm/charts/pull/13646), you might see
+      `spec.clusterIP: Invalid value` when trying to run the [`./gl-helmfile`](management_project_template.md#the-gitlab-ciyml-file)
+      command. To work around this, after overwriting the release values in `applications/ingress/values.yaml`,
+      you might need to overwrite all the occurrences of `omitClusterIP: false`, setting it to `omitClusterIP: true`.
+      Another approach,could be to collect these IPs by running `kubectl get services -n gitlab-managed-apps`
+      and then overwriting each `ClusterIP` that it complains about with the value you got from that command.
+
+    - Vault: This application introduces a breaking change from the chart we used in Helm v2 to the chart
+      used in Helm v3. So, the only way to integrate it with this Cluster Management Project is to actually uninstall this app and accept the
+      chart version proposed in `applications/vault/values.yaml`.
+
+1. After following all the previous steps, [run a pipeline manually](../../ci/pipelines/index.md#run-a-pipeline-manually)
+   and watch the `apply` job logs to see if any of your applications were successfully detected, installed, and whether they got any
+   unexpected updates.
+   
+   Some annotation checksums are expected to be updated, as well as this attribute:
+
+   ```diff
+   --- heritage: Tiller
+   +++ heritage: Tiller
+   ```
+
+After getting a successful pipeline, repeat these steps for any other deployed apps
+you want to manage with the Cluster Management Project.