Refactor CI script documentation

There are some ways to format script sections
to change how they display in job logs. These
should be documented separately, to allow for
more detailed examples (covering different OSes,
for example). We also need to split the before_script
and after_script details, as they work differently.

doc/.vale/gitlab/Acronyms.yml

@@ -25,6 +25,7 @@ exceptions:
   - CPU
   - CSS
   - CSV
+  - DAG
   - DAST
   - DNS
   - EKS

doc/ci/docker/using_docker_images.md

@@ -452,9 +452,9 @@ CI jobs:
    from `Dockerfile` that may be overridden in `.gitlab-ci.yml`)
 1. The runner attaches itself to a running container.
 1. The runner prepares a script (the combination of
-   [`before_script`](../yaml/README.md#before_script-and-after_script),
+   [`before_script`](../yaml/README.md#before_script),
    [`script`](../yaml/README.md#script),
-   and [`after_script`](../yaml/README.md#before_script-and-after_script)).
+   and [`after_script`](../yaml/README.md#after_script)).
 1. The runner sends the script to the container's shell STDIN and receives the
    output.
 

doc/ci/migration/jenkins.md

@@ -258,7 +258,7 @@ stages:
 ```
 
 Setting a step to be performed before and after any job can be done via the
-[`before_script` and `after_script` keywords](../yaml/README.md#before_script-and-after_script):
+[`before_script`](../yaml/README.md#before_script) and [`after_script`](../yaml/README.md#after_script) keywords:
 
 ```yaml
 default:

doc/ci/ssh_keys/README.md

@@ -93,7 +93,7 @@ to access it. This is where an SSH key pair comes in handy.
      # - git config --global user.name "User name"
    ```
 
-   The [`before_script`](../yaml/README.md#before_script-and-after_script) can be set globally
+   The [`before_script`](../yaml/README.md#before_script) can be set globally
    or per-job.
 
 1. Make sure the private server's [SSH host keys are verified](#verifying-the-ssh-host-keys).

doc/ci/variables/predefined_variables.md

@@ -69,7 +69,7 @@ Kubernetes-specific environment variables are detailed in the
 | `CI_JOB_MANUAL`                               | 8.12   | all    | The flag to indicate that job was manually started                                                                                                                                                                                                                                                                                                         |
 | `CI_JOB_NAME`                                 | 9.0    | 0.5    | The name of the job as defined in `.gitlab-ci.yml`                                                                                                                                                                                                                                                                                                         |
 | `CI_JOB_STAGE`                                | 9.0    | 0.5    | The name of the stage as defined in `.gitlab-ci.yml`                                                                                                                                                                                                                                                                                                       |
-| `CI_JOB_STATUS`                               | all    | 13.5   | The state of the job as each runner stage is executed. Use with [`after_script`](../yaml/README.md#before_script-and-after_script) where `CI_JOB_STATUS` can be either: `success`, `failed` or `canceled`.                                                                                                                                                                              |
+| `CI_JOB_STATUS`                               | all    | 13.5   | The state of the job as each runner stage is executed. Use with [`after_script`](../yaml/README.md#after_script) where `CI_JOB_STATUS` can be either: `success`, `failed` or `canceled`.                                                                                                                                                                              |
 | `CI_JOB_TOKEN`                                | 9.0    | 1.2    | Token used for authenticating with [a few API endpoints](../../api/README.md#gitlab-ci-job-token) and downloading [dependent repositories](../../user/project/new_ci_build_permissions_model.md#dependent-repositories). The token is valid as long as the job is running. |
 | `CI_JOB_JWT`                                  | 12.10  | all    | RS256 JSON web token that can be used for authenticating with third party systems that support JWT authentication, for example [HashiCorp's Vault](../secrets/index.md). |
 | `CI_JOB_URL`                                  | 11.1   | 0.5    | Job details URL                                                                                                                                                                                                                                                                                                                                            |

doc/ci/variables/where_variables_can_be_used.md

@@ -99,7 +99,7 @@ In the case of `after_script` scripts, they can:
 - Not use variables defined in `before_script` and `script`.
 
 These restrictions are because `after_script` scripts are executed in a
-[separated shell context](../yaml/README.md#before_script-and-after_script).
+[separated shell context](../yaml/README.md#after_script).
 
 ## Persisted variables
 

doc/ci/yaml/script.md

+---
+stage: Verify
+group: Continuous Integration
+info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers
+type: reference
+---
+
+# GitLab CI/CD script syntax
+
+You can use special syntax in [`script`](README.md#script) sections to:
+
+- [Split long commands](#split-long-commands) into multiline commands.
+- [Use color codes](#add-color-codes-to-script-output) to make job logs easier to review.
+- [Create custom collapsible sections](../pipelines/index.md#custom-collapsible-sections)
+  to simplify job log output.
+
+## Split long commands
+
+You can split long commands into multiline commands to improve readability with
+`|` (literal) and `>` (folded) [YAML multiline block scalar indicators](https://yaml-multiline.info/).
+
+CAUTION: **Warning:**
+If multiple commands are combined into one command string, only the last command's
+failure or success is reported.
+[Failures from earlier commands are ignored due to a bug](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/25394).
+To work around this, run each command as a separate `script:` item, or add an `exit 1`
+command to each command string.
+
+You can use the `|` (literal) YAML multiline block scalar indicator to write
+commands over multiple lines in the `script` section of a job description.
+Each line is treated as a separate command.
+Only the first command is repeated in the job log, but additional
+commands are still executed:
+
+```yaml
+job:
+  script:
+    - |
+      echo "First command line."
+      echo "Second command line."
+      echo "Third command line."
+```
+
+The example above renders in the job log as:
+
+```shell
+$ echo First command line # collapsed multiline command
+First command line
+Second command line.
+Third command line.
+```
+
+The `>` (folded) YAML multiline block scalar indicator treats empty lines between
+sections as the start of a new command:
+
+```yaml
+job:
+  script:
+    - >
+      echo "First command line
+      is split over two lines."
+
+      echo "Second command line."
+```
+
+This behaves similarly to multiline commands without the `>` or `|` block
+scalar indicators:
+
+```yaml
+job:
+  script:
+    - echo "First command line
+      is split over two lines."
+
+      echo "Second command line."
+```
+
+Both examples above render in the job log as:
+
+```shell
+$ echo First command line is split over two lines. # collapsed multiline command
+First command line is split over two lines.
+Second command line.
+```
+
+When you omit the `>` or `|` block scalar indicators, GitLab concatenates non-empty
+lines to form the command. Make sure the lines can run when concatenated.
+
+[Shell here documents](https://en.wikipedia.org/wiki/Here_document) work with the
+`|` and `>` operators as well. The example below transliterates lower case letters
+to upper case:
+
+```yaml
+job:
+  script:
+    - |
+      tr a-z A-Z << END_TEXT
+        one two three
+        four five six
+      END_TEXT
+```
+
+Results in:
+
+```shell
+$ tr a-z A-Z << END_TEXT # collapsed multiline command
+  ONE TWO THREE
+  FOUR FIVE SIX
+```
+
+## Add color codes to script output
+
+Script output can be colored using [ANSI escape codes](https://en.wikipedia.org/wiki/ANSI_escape_code#Colors),
+or by running commands or programs that output ANSI escape codes.
+
+For example, using [Bash with color codes](https://misc.flogisoft.com/bash/tip_colors_and_formatting):
+
+```yaml
+job:
+  script:
+    - echo -e "\e[31mThis text is red,\e[0m but this text isn't\e[31m however this text is red again."
+```
+
+You can define the color codes in Shell variables, or even [custom environment variables](../variables/README.md#custom-environment-variables),
+which makes the commands easier to read and reusable.
+
+For example, using the same example as above and variables defined in a `before_script`:
+
+```yaml
+job:
+  before_script:
+    - TXT_RED="\e[31m" && TXT_CLEAR="\e[0m"
+  script:
+    - echo -e "${TXT_RED}This text is red,${TXT_CLEAR} but this part isn't${TXT_RED} however this part is again."
+    - echo "This text is not colored"
+```
+
+Or with [PowerShell color codes](https://superuser.com/a/1259916):
+
+```yaml
+job:
+  before_script:
+    - $esc="$([char]27)"; $TXT_RED="$esc[31m"; $TXT_CLEAR="$esc[0m"
+  script:
+    - Write-Host $TXT_RED"This text is red,"$TXT_CLEAR" but this text isn't"$TXT_RED" however this text is red again."
+    - Write-Host "This text is not colored"
+```

doc/development/integrations/secure.md

@@ -46,12 +46,12 @@ Because the `script` entry can't be left empty, it must be set to the command th
 It is not possible to rely on the predefined `ENTRYPOINT` and `CMD` of the Docker image
 to perform the scan automatically, without passing any command.
 
-The [`before_script`](../../ci/yaml/README.md#before_script-and-after_script)
+The [`before_script`](../../ci/yaml/README.md#before_script)
 should not be used in the job definition because users may rely on this to prepare their projects before performing the scan.
 For instance, it is common practice to use `before_script` to install system libraries
 a particular project needs before performing SAST or Dependency Scanning.
 
-Similarly, [`after_script`](../../ci/yaml/README.md#before_script-and-after_script)
+Similarly, [`after_script`](../../ci/yaml/README.md#after_script)
 should not be used in the job definition, because it may be overridden by users.
 
 ### Stage

doc/user/application_security/container_scanning/index.md

@@ -148,8 +148,8 @@ variables:
 Version `3` of the `container_scanning` Docker image uses [`centos:centos8`](https://hub.docker.com/_/centos)
 as the new base. It also removes the use of the [start.sh](https://gitlab.com/gitlab-org/security-products/analyzers/klar/-/merge_requests/77)
 script and instead executes the analyzer by default. Any customizations made to the
-`container_scanning` job's [`before_script`](../../../ci/yaml/README.md#before_script-and-after_script)
-and [`after_script`](../../../ci/yaml/README.md#before_script-and-after_script)
+`container_scanning` job's [`before_script`](../../../ci/yaml/README.md#before_script)
+and [`after_script`](../../../ci/yaml/README.md#after_script)
 blocks may not work with the new version. To roll back to the previous [`alpine:3.11.3`](https://hub.docker.com/_/alpine)-based
 Docker image, you can specify the major version through the [`CS_MAJOR_VERSION`](#available-variables)
 variable.

doc/user/compliance/license_compliance/index.md

@@ -444,7 +444,7 @@ documentation for a list of settings that you can apply.
 The `license_scanning` job runs in a [Debian 10](https://www.debian.org/releases/buster/) Docker
 image. The supplied image ships with some build tools such as [CMake](https://cmake.org/) and [GCC](https://gcc.gnu.org/).
 However, not all project types are supported by default. To install additional tools needed to
-compile dependencies, use a [`before_script`](../../../ci/yaml/README.md#before_script-and-after_script)
+compile dependencies, use a [`before_script`](../../../ci/yaml/README.md#before_script)
 to install the necessary build tools using the [`apt`](https://wiki.debian.org/PackageManagementTools)
 package manager. For a comprehensive list, consult [the Conan documentation](https://docs.conan.io/en/latest/introduction.html#all-platforms-all-build-systems-and-compilers).
 

doc/user/gitlab_com/index.md

@@ -409,7 +409,7 @@ test:
 - For the beta release, we have included a set of software packages in
   the base VM image. If your CI job requires additional software that's
   not included in this list, then you will need to add installation
-  commands to [`before_script`](../../ci/yaml/README.md#before_script-and-after_script) or [`script`](../../ci/yaml/README.md#script) to install the required
+  commands to [`before_script`](../../ci/yaml/README.md#before_script) or [`script`](../../ci/yaml/README.md#script) to install the required
   software. Note that each job runs on a new VM instance, so the
   installation of additional software packages needs to be repeated for
   each job in your pipeline.