Merge branch 'docs/dep-scanning' into 'master'

Document all the available options for Dependency Scanning

Closes gitlab-ee#10121

See merge request gitlab-org/gitlab-ce!29347

doc/user/application_security/dependency_scanning/analyzers.md

+# Dependency Scanning Analyzers **[ULTIMATE]**
+
+Dependency Scanning relies on underlying third party tools that are wrapped into
+what we call "Analyzers". An analyzer is a
+[dedicated project](https://gitlab.com/gitlab-org/security-products/analyzers)
+that wraps a particular tool to:
+
+- Expose its detection logic.
+- Handle its execution.
+- Convert its output to the common format.
+
+This is achieved by implementing the [common API](https://gitlab.com/gitlab-org/security-products/analyzers/common).
+
+Dependency Scanning supports the following official analyzers:
+
+- [`bundler-audit`](https://gitlab.com/gitlab-org/security-products/analyzers/bundler-audit)
+- [`gemnasium`](https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium)
+- [`gemnasium-maven`](https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium-maven)
+- [`gemnasium-python`](https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium-python)
+- [`retire.js`](https://gitlab.com/gitlab-org/security-products/analyzers/retire.js)
+
+The analyzers are published as Docker images that Dependency Scanning will use
+to launch dedicated containers for each analysis.
+
+Dependency Scanning is pre-configured with a set of **default images** that are
+maintained by GitLab, but users can also integrate their own **custom images**.
+
+## Official default analyzers
+
+Any custom change to the official analyzers can be achieved by using an
+[environment variable in your `.gitlab-ci.yml`](index.md#customizing-the-dependency-scanning-settings).
+
+### Using a custom Docker mirror
+
+You can switch to a custom Docker registry that provides the official analyzer
+images under a different prefix. For instance, the following instructs Dependency
+Scanning to pull `my-docker-registry/gl-images/gemnasium`
+instead of `registry.gitlab.com/gitlab-org/security-products/analyzers/gemnasium`.
+In `.gitlab-ci.yml` define:
+
+```yaml
+include:
+  template: Dependency-Scanning.gitlab-ci.yml
+
+variables:
+  DS_ANALYZER_IMAGE_PREFIX: my-docker-registry/gl-images
+```
+
+This configuration requires that your custom registry provides images for all
+the official analyzers.
+
+### Selecting specific analyzers
+
+You can select the official analyzers you want to run. Here's how to enable
+`bundler-audit` and `gemnasium` while disabling all the other default ones.
+In `.gitlab-ci.yml` define:
+
+```yaml
+include:
+  template: Dependency-Scanning.gitlab-ci.yml
+
+variables:
+  DS_DEFAULT_ANALYZERS: "bundler-audit,gemnasium"
+```
+
+`bundler-audit` runs first. When merging the reports, Dependency Scanning will
+remove the duplicates and will keep the `bundler-audit` entries.
+
+### Disabling default analyzers
+
+Setting `DS_DEFAULT_ANALYZERS` to an empty string will disable all the official
+default analyzers. In `.gitlab-ci.yml` define:
+
+```yaml
+include:
+  template: Dependency-Scanning.gitlab-ci.yml
+
+variables:
+  DS_DEFAULT_ANALYZERS: ""
+```
+
+That's needed when one totally relies on [custom analyzers](#custom-analyzers).
+
+## Custom analyzers
+
+You can provide your own analyzers as a comma separated list of Docker images.
+Here's how to add `analyzers/nugget` and `analyzers/perl` to the default images.
+In `.gitlab-ci.yml` define:
+
+```yaml
+include:
+  template: Dependency-Scanning.gitlab-ci.yml
+
+variables:
+  DS_ANALYZER_IMAGES: "my-docker-registry/analyzers/nugget,amy-docker-registry/nalyzers/perl"
+```
+
+The values must be the full path to the container registry images,
+like what you would feed to the `docker pull` command.
+
+NOTE: **Note:**
+This configuration doesn't benefit from the integrated detection step. Dependency
+Scanning has to fetch and spawn each Docker image to establish whether the
+custom analyzer can scan the source code.
+
+## Analyzers data
+
+The following table lists the data available for each official analyzer.
+
+| Property \ Tool                       |      Gemnasium     |    bundler-audit   |     Retire.js      |
+|---------------------------------------|:------------------:|:------------------:|:------------------:|
+| Severity                              | 𐄂                  | ✓                  | ✓                  |
+| Title                                 | ✓                  | ✓                  | ✓                  |
+| File                                  | ✓                  | ⚠                  | ✓                  |
+| Start line                            | 𐄂                  | 𐄂                  | 𐄂                  |
+| End line                              | 𐄂                  | 𐄂                  | 𐄂                  |
+| External ID (e.g., CVE)               | ✓                  | ✓                  | ⚠                  |
+| URLs                                  | ✓                  | ✓                  | ✓                  |
+| Internal doc/explanation              | ✓                  | 𐄂                  | 𐄂                  |
+| Solution                              | ✓                  | ✓                  | 𐄂                  |
+| Confidence                            | 𐄂                  | 𐄂                  | 𐄂                  |
+| Affected item (e.g. class or package) | ✓                  | ✓                  | ✓                  |
+| Source code extract                   | 𐄂                  | 𐄂                  | 𐄂                  |
+| Internal ID                           | ✓                  | 𐄂                  | 𐄂                  |
+| Date                                  | ✓                  | 𐄂                  | 𐄂                  |
+| Credits                               | ✓                  | 𐄂                  | 𐄂                  |
+
+- ✓ => we have that data
+- ⚠ => we have that data but it's partially reliable, or we need to extract that data from unstructured content
+- 𐄂 => we don't have that data or it would need to develop specific or inefficient/unreliable logic to obtain it.
+
+The values provided by these tools are heterogeneous so they are sometimes
+normalized into common values (e.g., `severity`, `confidence`, etc).

doc/user/application_security/dependency_scanning/index.md

@@ -54,9 +54,22 @@ The following languages and dependency managers are supported.
 | Java ([Maven](https://maven.apache.org/))                                   | [gemnasium](https://gitlab.com/gitlab-org/security-products/gemnasium/general)                                                            |
 | PHP ([Composer](https://getcomposer.org/))                                  | [gemnasium](https://gitlab.com/gitlab-org/security-products/gemnasium/general)                                                            |
 
-Some scanners require to send a list of project dependencies to GitLab's central
-servers to check for vulnerabilities. To learn more about this or to disable it,
-refer to the [GitLab Dependency Scanning tool documentation](https://gitlab.com/gitlab-org/security-products/dependency-scanning#remote-checks).
+## Remote checks
+
+While some tools pull a local database to check vulnerabilities, some others
+like Gemnasium require sending data to GitLab central servers to analyze them:
+
+1. Gemnasium scans the dependencies of your project locally and sends a list of
+   packages to GitLab central servers.
+1. The servers return the list of known vulnerabilities for all versions of
+   these packages.
+1. The client picks up the relevant vulnerabilities by comparing with the versions
+   of the packages that are used by the project.
+
+The Gemnasium client does **NOT** send the exact package versions your project relies on.
+
+You can disable the remote checks by [using](#customizing-the-dependency-scanning-settings)
+the `DS_DISABLE_REMOTE_CHECKS` environment variable and setting it to `true`.
 
 ## Configuring Dependency Scanning
 
@@ -97,17 +110,10 @@ The report will be saved as a
 that you can later download and analyze. Due to implementation limitations, we
 always take the latest Dependency Scanning artifact available.
 
-Some security scanners require to send a list of project dependencies to GitLab
-central servers to check for vulnerabilities. To learn more about this or to
-disable it, check the
-[GitLab Dependency Scanning tool documentation](https://gitlab.com/gitlab-org/security-products/dependency-scanning#remote-checks).
-
 #### Customizing the Dependency Scanning settings
 
-The Dependency Scanning settings can be changed through environment variables by using the
+The Dependency Scanning settings can be changed through [environment variables](#available-variables) by using the
 [`variables`](../../../ci/yaml/README.md#variables) parameter in `.gitlab-ci.yml`.
-These variables are documented in the
-[Dependency Scanning tool documentation](https://gitlab.com/gitlab-org/security-products/dependency-scanning#settings).
 
 For example:
 
@@ -116,7 +122,7 @@ include:
   template: Dependency-Scanning.gitlab-ci.yml
 
 variables:
-  DEP_SCAN_DISABLE_REMOTE_CHECKS: true
+  DS_DISABLE_REMOTE_CHECKS: true
 ```
 
 Because template is [evaluated before](../../../ci/yaml/README.md#include) the pipeline
@@ -137,6 +143,24 @@ dependency_scanning:
     CI_DEBUG_TRACE: "true"
 ```
 
+#### Available variables
+
+Dependency Scanning can be [configured](#customizing-the-dependency-scanning-settings)
+using environment variables.
+
+| Environment variable                    | Function |
+|--------------------------------         |----------|
+| `DS_ANALYZER_IMAGES`                    | Comma separated list of custom images. The official default images are still enabled. Read more about [customizing analyzers](analyzers.md). |
+| `DS_ANALYZER_IMAGE_PREFIX`              | Override the name of the Docker registry providing the official default images (proxy). Read more about [customizing analyzers](analyzers.md). |
+| `DS_ANALYZER_IMAGE_TAG`                 | Override the Docker tag of the official default images. Read more about [customizing analyzers](analyzers.md). |
+| `DS_DEFAULT_ANALYZERS`                  | Override the names of the official default images. Read more about [customizing analyzers](analyzers.md). |
+| `DS_DISABLE_REMOTE_CHECKS`              | Do not send any data to GitLab. Used in the [Gemnasium analyzer](#remote-checks). |
+| `DS_PULL_ANALYZER_IMAGES`               | Pull the images from the Docker registry (set to `0` to disable). |
+| `DS_EXCLUDED_PATHS`                     | Exclude vulnerabilities from output based on the paths. A comma-separated list of patterns. Patterns can be globs, file or folder paths. Parent directories will also match patterns. |
+| `DS_DOCKER_CLIENT_NEGOTIATION_TIMEOUT`  | Time limit for Docker client negotiation. Timeouts are parsed using Go's [`ParseDuration`](https://golang.org/pkg/time/#ParseDuration). Valid time units are `ns`, `us` (or `µs`), `ms`, `s`, `m`, `h`. For example, `300ms`, `1.5h`, or `2h45m`. |
+| `DS_PULL_ANALYZER_IMAGE_TIMEOUT`        | Time limit when pulling the image of an analyzer. Timeouts are parsed using Go's [`ParseDuration`](https://golang.org/pkg/time/#ParseDuration). Valid time units are `ns`, `us` (or `µs`), `ms`, `s`, `m`, `h`. For example, `300ms`, `1.5h`, or `2h45m`. |
+| `DS_RUN_ANALYZER_TIMEOUT`               | Time limit when running an analyzer. Timeouts are parsed using Go's [`ParseDuration`](https://golang.org/pkg/time/#ParseDuration). Valid time units are `ns`, `us` (or `µs`), `ms`, `s`, `m`, `h`. For example, `300ms`, `1.5h`, or `2h45m`. |
+
 ### Manual job definition for GitLab 11.5 and later
 
 For GitLab 11.5 and GitLab Runner 11.5 and later, the following `dependency_scanning`
@@ -171,7 +195,7 @@ dependency_scanning:
       dependency_scanning: gl-dependency-scanning-report.json
 ```
 
-You can supply many other [settings variables](https://gitlab.com/gitlab-org/security-products/dependency-scanning#settings)
+You can supply many other [settings variables](#available-variables)
 via `docker run --env` to customize your job execution.
 
 ### Manual job definition for GitLab 11.4 and earlier (deprecated)
@@ -388,6 +412,10 @@ supported by Gemnasium.
 
 To see the generated dependency list, navigate to your project's **Project > Dependency List**.
 
+## Versioning and release process
+
+Please check the [Release Process documentation](https://gitlab.com/gitlab-org/security-products/release/blob/master/docs/release_process.md).
+
 ## Contributing to the vulnerability database
 
 You can search the [gemnasium-db](https://gitlab.com/gitlab-org/security-products/gemnasium-db) project