Merge branch 'selhorn-docs-runner-build-cloud-move' into 'master'

Docs: Moved content in prep for 14.1 release

See merge request gitlab-org/gitlab!65193

doc/ci/runners/build_cloud/linux_build_cloud.md

+---
+stage: Verify
+group: Runner
+info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments
+---
+
+# Linux shared runners
+
+Linux shared runners on GitLab.com run in autoscale mode and are powered by Google Cloud Platform.
+
+Autoscaling means reduced queue times to spin up CI/CD jobs, and isolated VMs for each job, thus maximizing security. These shared runners are available for users and customers on GitLab.com.
+
+GitLab offers Ultimate tier capabilities and included CI/CD minutes per group per month for our [Open Source](https://about.gitlab.com/solutions/open-source/join/), [Education](https://about.gitlab.com/solutions/education/), and [Startups](https://about.gitlab.com/solutions/startups/) programs. For private projects, GitLab offers various [plans](https://about.gitlab.com/pricing/), starting with a Free tier.
+
+All your CI/CD jobs run on [n1-standard-1 instances](https://cloud.google.com/compute/docs/machine-types) with 3.75GB of RAM, CoreOS and the latest Docker Engine
+installed. Instances provide 1 vCPU and 25GB of HDD disk space. The default
+region of the VMs is US East1.
+Each instance is used only for one job, this ensures any sensitive data left on the system can't be accessed by other people their CI jobs.
+
+The `gitlab-shared-runners-manager-X.gitlab.com` fleet of runners are dedicated for GitLab projects as well as community forks of them. They use a slightly larger machine type (n1-standard-2) and have a bigger SSD disk size. They don't run untagged jobs and unlike the general fleet of shared runners, the instances are re-used up to 40 times.
+
+Jobs handled by the shared runners on GitLab.com (`shared-runners-manager-X.gitlab.com`),
+**time out after 3 hours**, regardless of the timeout configured in a
+project. Check the issues [4010](https://gitlab.com/gitlab-com/infrastructure/-/issues/4010) and [4070](https://gitlab.com/gitlab-com/infrastructure/-/issues/4070) for the reference.
+
+Below are the shared runners settings.
+
+| Setting                               | GitLab.com                                        | Default    |
+| -----------                           | -----------------                                 | ---------- |
+| [GitLab Runner](https://gitlab.com/gitlab-org/gitlab-runner) | [Runner versions dashboard](https://dashboards.gitlab.net/d/ci-runners-deployment/ci-runners-deployment-overview?orgId=1&refresh=1m) | - |
+| Executor                              | `docker+machine`                                  | -          |
+| Default Docker image                  | `ruby:2.5`                                        | -          |
+| `privileged` (run [Docker in Docker](https://hub.docker.com/_/docker/)) | `true`          | `false`    |
+
+## Pre-clone script
+
+Linux shared runners on GitLab.com provide a way to run commands in a CI
+job before the runner attempts to run `git init` and `git fetch` to
+download a GitLab repository. The
+[`pre_clone_script`](https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-runners-section)
+can be used for:
+
+- Seeding the build directory with repository data
+- Sending a request to a server
+- Downloading assets from a CDN
+- Any other commands that must run before the `git init`
+
+To use this feature, define a [CI/CD variable](../../../ci/variables/index.md#custom-cicd-variables) called
+`CI_PRE_CLONE_SCRIPT` that contains a bash script.
+
+[This example](../../../development/pipelines.md#pre-clone-step)
+demonstrates how you might use a pre-clone step to seed the build
+directory.
+
+NOTE:
+The `CI_PRE_CLONE_SCRIPT` variable does not work on Windows runners.
+
+## `config.toml`
+
+The full contents of our `config.toml` are:
+
+NOTE:
+Settings that are not public are shown as `X`.
+
+**Google Cloud Platform**
+
+```toml
+concurrent = X
+check_interval = 1
+metrics_server = "X"
+sentry_dsn = "X"
+
+[[runners]]
+  name = "docker-auto-scale"
+  request_concurrency = X
+  url = "https://gitlab.com/"
+  token = "SHARED_RUNNER_TOKEN"
+  pre_clone_script = "eval \"$CI_PRE_CLONE_SCRIPT\""
+  executor = "docker+machine"
+  environment = [
+    "DOCKER_DRIVER=overlay2",
+    "DOCKER_TLS_CERTDIR="
+  ]
+  limit = X
+  [runners.docker]
+    image = "ruby:2.5"
+    privileged = true
+    volumes = [
+      "/certs/client",
+      "/dummy-sys-class-dmi-id:/sys/class/dmi/id:ro" # Make kaniko builds work on GCP.
+    ]
+  [runners.machine]
+    IdleCount = 50
+    IdleTime = 3600
+    MaxBuilds = 1 # For security reasons we delete the VM after job has finished so it's not reused.
+    MachineName = "srm-%s"
+    MachineDriver = "google"
+    MachineOptions = [
+      "google-project=PROJECT",
+      "google-disk-size=25",
+      "google-machine-type=n1-standard-1",
+      "google-username=core",
+      "google-tags=gitlab-com,srm",
+      "google-use-internal-ip",
+      "google-zone=us-east1-d",
+      "engine-opt=mtu=1460", # Set MTU for container interface, for more information check https://gitlab.com/gitlab-org/gitlab-runner/-/issues/3214#note_82892928
+      "google-machine-image=PROJECT/global/images/IMAGE",
+      "engine-opt=ipv6", # This will create IPv6 interfaces in the containers.
+      "engine-opt=fixed-cidr-v6=fc00::/7",
+      "google-operation-backoff-initial-interval=2" # Custom flag from forked docker-machine, for more information check https://github.com/docker/machine/pull/4600
+    ]
+    [[runners.machine.autoscaling]]
+      Periods = ["* * * * * sat,sun *"]
+      Timezone = "UTC"
+      IdleCount = 70
+      IdleTime = 3600
+    [[runners.machine.autoscaling]]
+      Periods = ["* 30-59 3 * * * *", "* 0-30 4 * * * *"]
+      Timezone = "UTC"
+      IdleCount = 700
+      IdleTime = 3600
+  [runners.cache]
+    Type = "gcs"
+    Shared = true
+    [runners.cache.gcs]
+      CredentialsFile = "/path/to/file"
+      BucketName = "bucket-name"
+```

doc/ci/runners/build_cloud/windows_build_cloud.md

+---
+stage: Verify
+group: Runner
+info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments
+---
+
+# Windows shared runners (beta)
+
+The Windows shared runners are in [beta](https://about.gitlab.com/handbook/product/gitlab-the-product/#beta)
+and shouldn't be used for production workloads.
+
+During this beta period, the [shared runner pipeline quota](../../../user/admin_area/settings/continuous_integration.md#shared-runners-pipeline-minutes-quota)
+applies for groups and projects in the same manner as Linux runners. This may
+change when the beta period ends, as discussed in this [related issue](https://gitlab.com/gitlab-org/gitlab/-/issues/30834).
+
+Windows shared runners on GitLab.com autoscale by launching virtual machines on
+the Google Cloud Platform. This solution uses an
+[autoscaling driver](https://gitlab.com/gitlab-org/ci-cd/custom-executor-drivers/autoscaler/tree/master/docs/readme.md)
+developed by GitLab for the [custom executor](https://docs.gitlab.com/runner/executors/custom.html).
+Windows shared runners execute your CI/CD jobs on `n1-standard-2` instances with
+2 vCPUs and 7.5 GB RAM. You can find a full list of available Windows packages in
+the [package documentation](https://gitlab.com/gitlab-org/ci-cd/shared-runners/images/gcp/windows-containers/blob/master/cookbooks/preinstalled-software/README.md).
+
+We want to keep iterating to get Windows shared runners in a stable state and
+[generally available](https://about.gitlab.com/handbook/product/gitlab-the-product/#generally-available-ga).
+You can follow our work towards this goal in the
+[related epic](https://gitlab.com/groups/gitlab-org/-/epics/2162).
+
+## Configuration
+
+The full contents of our `config.toml` are:
+
+NOTE:
+Settings that aren't public are shown as `X`.
+
+```toml
+concurrent = X
+check_interval = 3
+
+[[runners]]
+  name = "windows-runner"
+  url = "https://gitlab.com/"
+  token = "TOKEN"
+  executor = "custom"
+  builds_dir = "C:\\GitLab-Runner\\builds"
+  cache_dir = "C:\\GitLab-Runner\\cache"
+  shell  = "powershell"
+  [runners.custom]
+    config_exec = "C:\\GitLab-Runner\\autoscaler\\autoscaler.exe"
+    config_args = ["--config", "C:\\GitLab-Runner\\autoscaler\\config.toml", "custom", "config"]
+    prepare_exec = "C:\\GitLab-Runner\\autoscaler\\autoscaler.exe"
+    prepare_args = ["--config", "C:\\GitLab-Runner\\autoscaler\\config.toml", "custom", "prepare"]
+    run_exec = "C:\\GitLab-Runner\\autoscaler\\autoscaler.exe"
+    run_args = ["--config", "C:\\GitLab-Runner\\autoscaler\\config.toml", "custom", "run"]
+    cleanup_exec = "C:\\GitLab-Runner\\autoscaler\\autoscaler.exe"
+    cleanup_args = ["--config", "C:\\GitLab-Runner\\autoscaler\\config.toml", "custom", "cleanup"]
+```
+
+The full contents of our `autoscaler/config.toml` are:
+
+```toml
+Provider = "gcp"
+Executor = "winrm"
+OS = "windows"
+LogLevel = "info"
+LogFormat = "text"
+LogFile = "C:\\GitLab-Runner\\autoscaler\\autoscaler.log"
+VMTag = "windows"
+
+[GCP]
+  ServiceAccountFile = "PATH"
+  Project = "some-project-df9323"
+  Zone = "us-east1-c"
+  MachineType = "n1-standard-2"
+  Image = "IMAGE"
+  DiskSize = 50
+  DiskType = "pd-standard"
+  Subnetwork = "default"
+  Network = "default"
+  Tags = ["TAGS"]
+  Username = "gitlab_runner"
+
+[WinRM]
+  MaximumTimeout = 3600
+  ExecutionMaxRetries = 0
+
+[ProviderCache]
+  Enabled = true
+  Directory = "C:\\GitLab-Runner\\autoscaler\\machines"
+```
+
+## Example
+
+Below is a simple `.gitlab-ci.yml` file to show how to start using the
+Windows shared runners:
+
+```yaml
+.shared_windows_runners:
+  tags:
+    - shared-windows
+    - windows
+    - windows-1809
+
+stages:
+  - build
+  - test
+
+before_script:
+ - Set-Variable -Name "time" -Value (date -Format "%H:%m")
+ - echo ${time}
+ - echo "started by ${GITLAB_USER_NAME}"
+
+build:
+  extends:
+    - .shared_windows_runners
+  stage: build
+  script:
+    - echo "running scripts in the build job"
+
+test:
+  extends:
+    - .shared_windows_runners
+  stage: test
+  script:
+    - echo "running scripts in the test job"
+```
+
+## Limitations and known issues
+
+- All the limitations mentioned in our [beta
+  definition](https://about.gitlab.com/handbook/product/#beta).
+- The average provisioning time for a new Windows VM is 5 minutes.
+  This means that you may notice slower build start times
+  on the Windows shared runner fleet during the beta. In a future
+  release we intend to update the autoscaler to enable
+  the pre-provisioning of virtual machines. This is intended to significantly reduce
+  the time it takes to provision a VM on the Windows fleet. You can
+  follow along in the [related issue](https://gitlab.com/gitlab-org/ci-cd/custom-executor-drivers/autoscaler/-/issues/32).
+- The Windows shared runner fleet may be unavailable occasionally
+  for maintenance or updates.
+- The Windows shared runner virtual machine instances do not use the
+  GitLab Docker executor. This means that you can't specify
+  [`image`](../../../ci/yaml/index.md#image) or [`services`](../../../ci/yaml/index.md#services) in
+  your pipeline configuration.
+- For the beta release, we have included a set of software packages in
+  the base VM image. If your CI job requires additional software that's
+  not included in this list, then you must add installation
+  commands to [`before_script`](../../../ci/yaml/index.md#before_script) or [`script`](../../../ci/yaml/index.md#script) to install the required
+  software. Note that each job runs on a new VM instance, so the
+  installation of additional software packages needs to be repeated for
+  each job in your pipeline.
+- The job may stay in a pending state for longer than the
+  Linux shared runners.
+- There is the possibility that we introduce breaking changes which will
+  require updates to pipelines that are using the Windows shared runner
+  fleet.

doc/development/pipelines.md

@@ -585,7 +585,7 @@ several reasons:
 - It significantly reduces load on the file server, as smaller deltas mean less time spent in `git pack-objects`.
 
 The pre-clone step works by using the `CI_PRE_CLONE_SCRIPT` variable
-[defined by GitLab.com shared runners](../ci/runners/index.md#pre-clone-script).
+[defined by GitLab.com shared runners](../ci/runners/build_cloud/linux_build_cloud.md#pre-clone-script).
 
 The `CI_PRE_CLONE_SCRIPT` is currently defined as a project CI/CD variable:
 

doc/user/project/merge_requests/load_performance_testing.md

@@ -93,7 +93,7 @@ template that is included with GitLab.
 NOTE:
 For large scale k6 tests you need to ensure the GitLab Runner instance performing the actual
 test is able to handle running the test. Refer to [k6's guidance](https://k6.io/docs/testing-guides/running-large-tests#hardware-considerations)
-for spec details. The [default shared GitLab.com runners](../../../ci/runners/index.md#linux-shared-runners)
+for spec details. The [default shared GitLab.com runners](../../../ci/runners/build_cloud/linux_build_cloud.md)
 likely have insufficient specs to handle most large k6 tests.
 
 This template runs the