Merge branch 'bwill/default-branch-image-fingerprint' into 'master'

Add default_branch_image to container scanning location

See merge request gitlab-org/gitlab!73486

config/feature_flags/development/improved_container_scan_matching.yml

+---
+name: improved_container_scan_matching
+introduced_by_url: https://gitlab.com/gitlab-org/gitlab/-/merge_requests/73486
+rollout_issue_url: https://gitlab.com/gitlab-org/gitlab/-/issues/344534
+milestone: '14.6'
+type: development
+group: group::container security
+default_enabled: false

doc/user/application_security/container_scanning/index.md

@@ -134,6 +134,7 @@ You can [configure](#customizing-the-container-scanning-settings) analyzers by u
 | `CI_APPLICATION_REPOSITORY`    | `$CI_REGISTRY_IMAGE/$CI_COMMIT_REF_SLUG` | Docker repository URL for the image to be scanned. | All |
 | `CI_APPLICATION_TAG`           | `$CI_COMMIT_SHA` | Docker repository tag for the image to be scanned. | All |
 | `CS_ANALYZER_IMAGE`            | `registry.gitlab.com/security-products/container-scanning:4` | Docker image of the analyzer. | All |
+| `CS_DEFAULT_BRANCH_IMAGE`      | `""` | The name of the `DOCKER_IMAGE` on the default branch. See [Setting the default branch image](#setting-the-default-branch-image) for more details. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/338877) in GitLab 14.5. | All |
 | `CS_DOCKER_INSECURE`           | `"false"`     | Allow access to secure Docker registries using HTTPS without validating the certificates. | All |
 | `CS_REGISTRY_INSECURE`         | `"false"`     | Allow access to insecure registries (HTTP only). Should only be set to `true` when testing the image locally. Works with all scanners, but the registry must listen on port `80/tcp` for Trivy to work. | All |
 | `CS_SEVERITY_THRESHOLD`        | `UNKNOWN`     | Severity level threshold. The scanner outputs vulnerabilities with severity level higher than or equal to this threshold. Supported levels are Unknown, Low, Medium, High, and Critical. | Trivy |
@@ -229,6 +230,51 @@ Prior to the GitLab 14.0 release, any variable defined under the scope `containe
 considered for scanners other than Clair. In GitLab 14.0 and later, all variables can be defined
 either as a global variable or under `container_scanning`.
 
+### Setting the default branch image
+
+> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/338877) in GitLab 14.5.
+
+By default, container scanning assumes that the image naming convention stores any branch-specific
+identifiers in the image tag rather than the image name. When the image name differs between the
+default branch and the non-default branch, previously-detected vulnerabilities show up as newly
+detected in merge requests.
+
+When the same image has different names on the default branch and a non-default branch, you can use
+the `CS_DEFAULT_BRANCH_IMAGE` variable to indicate what that image's name is on the default branch.
+GitLab then correctly determines if a vulnerability already exists when running scans on non-default
+branches.
+
+As an example, suppose the following:
+
+- Non-default branches publish images with the naming convention
+  `$CI_REGISTRY_IMAGE/$CI_COMMIT_BRANCH:$CI_COMMIT_SHA`.
+- The default branch publishes images with the naming convention
+  `$CI_REGISTRY_IMAGE:$CI_COMMIT_SHA`.
+
+In this example, you can use the following CI/CD configuration to ensure that vulnerabilities aren't
+duplicated:
+
+```yaml
+include:
+  - template: Security/Container-Scanning.gitlab-ci.yml
+
+container_scanning:
+  variables:
+    CS_DEFAULT_BRANCH_IMAGE: $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA
+  before_script:
+    - export DOCKER_IMAGE="$CI_REGISTRY_IMAGE/$CI_COMMIT_BRANCH:$CI_COMMIT_SHA"
+    - |
+      if [ "$CI_COMMIT_BRANCH" == "$CI_DEFAULT_BRANCH" ]; then
+        export DOCKER_IMAGE="$CI_REGISTRY_IMAGE:$CI_COMMIT_SHA"
+      fi
+```
+
+`CS_DEFAULT_BRANCH_IMAGE` should remain the same for a given `DOCKER_IMAGE`. If it changes, then a
+duplicate set of vulnerabilities are created, which must be manually dismissed.
+
+When using [Auto DevOps](../../../topics/autodevops/index.md), `CS_DEFAULT_BRANCH_IMAGE` is
+automatically set to `$CI_REGISTRY_IMAGE/$CI_DEFAULT_BRANCH:$CI_APPLICATION_TAG`.
+
 ### Using a custom SSL CA certificate authority
 
 You can use the `ADDITIONAL_CA_CERT_BUNDLE` CI/CD variable to configure a custom SSL CA certificate authority, which is used to verify the peer when fetching Docker images from a registry which uses HTTPS. The `ADDITIONAL_CA_CERT_BUNDLE` value should contain the [text representation of the X.509 PEM public-key certificate](https://tools.ietf.org/html/rfc7468#section-5.1). For example, to configure this value in the `.gitlab-ci.yml` file, use the following:
@@ -500,7 +546,7 @@ Here's an example container scanning report:
 
 ```json-doc
 {
-  "version": "3.0.0",
+  "version": "14.0.0",
   "vulnerabilities": [
     {
       "id": "df52bc8ce9a2ae56bbcb0c4ecda62123fbd6f69b",
@@ -522,7 +568,8 @@ Here's an example container scanning report:
           "version": "1.4.8"
         },
         "operating_system": "debian:9.4",
-        "image": "registry.gitlab.com/gitlab-org/security-products/dast/webgoat-8.0@sha256:bc09fe2e0721dfaeee79364115aeedf2174cce0947b9ae5fe7c33312ee019a4e"
+        "image": "registry.gitlab.com/gitlab-org/security-products/dast/webgoat-8.0@sha256:bc09fe2e0721dfaeee79364115aeedf2174cce0947b9ae5fe7c33312ee019a4e",
+        "default_branch_image": "registry.gitlab.com/gitlab-org/security-products/dast/webgoat-8.0:latest"
       },
       "identifiers": [
         {

ee/lib/gitlab/ci/parsers/security/container_scanning.rb

@@ -12,7 +12,22 @@ module Gitlab
               image: location_data['image'],
               operating_system: location_data['operating_system'],
               package_name: location_data.dig('dependency', 'package', 'name'),
-              package_version: location_data.dig('dependency', 'version'))
+              package_version: location_data.dig('dependency', 'version'),
+              default_branch_image: default_branch_image(location_data),
+              improved_container_scan_matching_enabled: improved_container_scan_matching_enabled?
+            )
+          end
+
+          def default_branch_image(location_data)
+            return unless improved_container_scan_matching_enabled?
+
+            return if @report.pipeline.default_branch?
+
+            location_data['default_branch_image']
+          end
+
+          def improved_container_scan_matching_enabled?
+            Feature.enabled?(:improved_container_scan_matching, @report.pipeline.project, default_enabled: :yaml)
           end
         end
       end

ee/lib/gitlab/ci/reports/security/locations/container_scanning.rb

@@ -6,28 +6,49 @@ module Gitlab
       module Security
         module Locations
           class ContainerScanning < Base
+            attr_reader :default_branch_image
             attr_reader :image
             attr_reader :operating_system
             attr_reader :package_name
             attr_reader :package_version
 
-            def initialize(image:, operating_system:, package_name: nil, package_version: nil)
+            def initialize(
+              image:,
+              operating_system:,
+              package_name: nil,
+              package_version: nil,
+              default_branch_image: nil,
+              improved_container_scan_matching_enabled: false
+            )
               @image = image
               @operating_system = operating_system
               @package_name = package_name
               @package_version = package_version
+              @default_branch_image = default_branch_image
+              @improved_container_scan_matching_enabled = improved_container_scan_matching_enabled
             end
 
             def fingerprint_data
               "#{docker_image_name_without_tag}:#{package_name}"
             end
 
+            def improved_container_scan_matching_enabled?
+              @improved_container_scan_matching_enabled
+            end
+
             private
 
             def docker_image_name_without_tag
-              base_name, version = image.split(':')
+              if improved_container_scan_matching_enabled?
+                image_name = default_branch_image.presence || image
+                base_name, _, version = image_name.rpartition(':')
 
-              return image if version_semver_like?(version)
+                return image_name if version_semver_like?(version)
+              else
+                base_name, version = image.split(':')
+
+                return image if version_semver_like?(version)
+              end
 
               base_name
             end

ee/spec/fixtures/security_reports/master/gl-container-scanning-report.json

@@ -21,7 +21,8 @@
           "version": "2.24-11+deb9u3"
         },
         "operating_system": "debian:9",
-        "image": "registry.gitlab.com/gitlab-org/security-products/dast/webgoat-8.0@sha256:bc09fe2e0721dfaeee79364115aeedf2174cce0947b9ae5fe7c33312ee019a4e"
+        "image": "registry.gitlab.com/gitlab-org/security-products/dast/webgoat-8.0@sha256:bc09fe2e0721dfaeee79364115aeedf2174cce0947b9ae5fe7c33312ee019a4e",
+        "default_branch_image": "registry.gitlab.com/gitlab-org/security-products/dast/webgoat-8.0:latest"
       },
       "identifiers": [
         {
@@ -57,7 +58,8 @@
           "version": "2.24-11+deb9u3"
         },
         "operating_system": "debian:9",
-        "image": "registry.gitlab.com/gitlab-org/security-products/dast/webgoat-8.0@sha256:bc09fe2e0721dfaeee79364115aeedf2174cce0947b9ae5fe7c33312ee019a4e"
+        "image": "registry.gitlab.com/gitlab-org/security-products/dast/webgoat-8.0@sha256:bc09fe2e0721dfaeee79364115aeedf2174cce0947b9ae5fe7c33312ee019a4e",
+        "default_branch_image": "registry.gitlab.com/gitlab-org/security-products/dast/webgoat-8.0:latest"
       },
       "identifiers": [
         {
@@ -92,7 +94,8 @@
           "version": "2.24-11+deb9u3"
         },
         "operating_system": "debian:9",
-        "image": "registry.gitlab.com/gitlab-org/security-products/dast/webgoat-8.0@sha256:bc09fe2e0721dfaeee79364115aeedf2174cce0947b9ae5fe7c33312ee019a4e"
+        "image": "registry.gitlab.com/gitlab-org/security-products/dast/webgoat-8.0@sha256:bc09fe2e0721dfaeee79364115aeedf2174cce0947b9ae5fe7c33312ee019a4e",
+        "default_branch_image": "registry.gitlab.com/gitlab-org/security-products/dast/webgoat-8.0:latest"
       },
       "identifiers": [
         {
@@ -127,7 +130,8 @@
           "version": "2.24-11+deb9u3"
         },
         "operating_system": "debian:9",
-        "image": "registry.gitlab.com/gitlab-org/security-products/dast/webgoat-8.0@sha256:bc09fe2e0721dfaeee79364115aeedf2174cce0947b9ae5fe7c33312ee019a4e"
+        "image": "registry.gitlab.com/gitlab-org/security-products/dast/webgoat-8.0@sha256:bc09fe2e0721dfaeee79364115aeedf2174cce0947b9ae5fe7c33312ee019a4e",
+        "default_branch_image": "registry.gitlab.com/gitlab-org/security-products/dast/webgoat-8.0:latest"
       },
       "identifiers": [
         {
@@ -162,7 +166,8 @@
           "version": "0.168-1"
         },
         "operating_system": "debian:9",
-        "image": "registry.gitlab.com/gitlab-org/security-products/dast/webgoat-8.0@sha256:bc09fe2e0721dfaeee79364115aeedf2174cce0947b9ae5fe7c33312ee019a4e"
+        "image": "registry.gitlab.com/gitlab-org/security-products/dast/webgoat-8.0@sha256:bc09fe2e0721dfaeee79364115aeedf2174cce0947b9ae5fe7c33312ee019a4e",
+        "default_branch_image": "registry.gitlab.com/gitlab-org/security-products/dast/webgoat-8.0:latest"
       },
       "identifiers": [
         {
@@ -197,7 +202,8 @@
           "version": "2.24-11+deb9u3"
         },
         "operating_system": "debian:9",
-        "image": "registry.gitlab.com/gitlab-org/security-products/dast/webgoat-8.0@sha256:bc09fe2e0721dfaeee79364115aeedf2174cce0947b9ae5fe7c33312ee019a4e"
+        "image": "registry.gitlab.com/gitlab-org/security-products/dast/webgoat-8.0@sha256:bc09fe2e0721dfaeee79364115aeedf2174cce0947b9ae5fe7c33312ee019a4e",
+        "default_branch_image": "registry.gitlab.com/gitlab-org/security-products/dast/webgoat-8.0:latest"
       },
       "identifiers": [
         {
@@ -232,7 +238,8 @@
           "version": "3.3-1"
         },
         "operating_system": "debian:9",
-        "image": "registry.gitlab.com/gitlab-org/security-products/dast/webgoat-8.0@sha256:bc09fe2e0721dfaeee79364115aeedf2174cce0947b9ae5fe7c33312ee019a4e"
+        "image": "registry.gitlab.com/gitlab-org/security-products/dast/webgoat-8.0@sha256:bc09fe2e0721dfaeee79364115aeedf2174cce0947b9ae5fe7c33312ee019a4e",
+        "default_branch_image": "registry.gitlab.com/gitlab-org/security-products/dast/webgoat-8.0:latest"
       },
       "identifiers": [
         {
@@ -268,7 +275,8 @@
           "version": "5.24.1-3+deb9u3"
         },
         "operating_system": "debian:9",
-        "image": "registry.gitlab.com/gitlab-org/security-products/dast/webgoat-8.0@sha256:bc09fe2e0721dfaeee79364115aeedf2174cce0947b9ae5fe7c33312ee019a4e"
+        "image": "registry.gitlab.com/gitlab-org/security-products/dast/webgoat-8.0@sha256:bc09fe2e0721dfaeee79364115aeedf2174cce0947b9ae5fe7c33312ee019a4e",
+        "default_branch_image": "registry.gitlab.com/gitlab-org/security-products/dast/webgoat-8.0:latest"
       },
       "identifiers": [
         {
@@ -299,4 +307,4 @@
     "type": "container_scanning",
     "status": "success"
   }
-}
+}
\ No newline at end of file

ee/spec/lib/gitlab/ci/parsers/security/container_scanning_spec.rb

@@ -3,18 +3,16 @@
 require 'spec_helper'
 
 RSpec.describe Gitlab::Ci::Parsers::Security::ContainerScanning do
-  let(:project) { artifact.project }
-  let(:pipeline) { artifact.job.pipeline }
+  let(:project) { create(:project, :repository) }
+  let(:current_branch) { project.default_branch }
+  let(:pipeline) { create(:ci_pipeline, ref: current_branch, project: project) }
+  let(:job) { create(:ci_build, pipeline: pipeline)}
+  let(:artifact) { create(:ee_ci_job_artifact, :container_scanning, job: job) }
   let(:report) { Gitlab::Ci::Reports::Security::Report.new(artifact.file_type, pipeline, 2.weeks.ago) }
+  let(:image) { 'registry.gitlab.com/gitlab-org/security-products/dast/webgoat-8.0@sha256:bc09fe2e0721dfaeee79364115aeedf2174cce0947b9ae5fe7c33312ee019a4e' }
+  let(:default_branch_image) { 'registry.gitlab.com/gitlab-org/security-products/dast/webgoat-8.0:latest' }
 
-  before do
-    artifact.each_blob { |blob| described_class.parse!(blob, report) }
-  end
-
-  describe '#parse!' do
-    let(:artifact) { create(:ee_ci_job_artifact, :container_scanning) }
-    let(:image) { 'registry.gitlab.com/gitlab-org/security-products/dast/webgoat-8.0@sha256:bc09fe2e0721dfaeee79364115aeedf2174cce0947b9ae5fe7c33312ee019a4e' }
-
+  shared_examples 'report' do
     it "parses all identifiers and findings for unapproved vulnerabilities" do
       expect(report.findings.length).to eq(8)
       expect(report.identifiers.length).to eq(8)
@@ -42,4 +40,66 @@ RSpec.describe Gitlab::Ci::Parsers::Security::ContainerScanning do
       expect(Gitlab::Json.parse(report.findings.first.raw_metadata).dig('location', 'image')).to eq(image)
     end
   end
+
+  describe '#parse!' do
+    context 'when improved_container_scan_matching is disabled' do
+      before do
+        stub_feature_flags(improved_container_scan_matching: false)
+        artifact.each_blob { |blob| described_class.parse!(blob, report) }
+      end
+
+      it_behaves_like 'report'
+
+      context 'when not on default branch' do
+        let(:current_branch) { 'not-default' }
+
+        it 'does not include default_branch_image' do
+          location = report.findings.first.location
+
+          expect(location).to be_a(::Gitlab::Ci::Reports::Security::Locations::ContainerScanning)
+          expect(location).to have_attributes(
+            default_branch_image: nil,
+            improved_container_scan_matching_enabled?: false
+          )
+        end
+      end
+    end
+
+    context 'when improved_container_scan_matching is enabled' do
+      before do
+        stub_feature_flags(improved_container_scan_matching: true)
+        artifact.each_blob { |blob| described_class.parse!(blob, report) }
+      end
+
+      it_behaves_like 'report'
+
+      context 'when on default branch' do
+        let(:current_branch) { project.default_branch }
+
+        it 'does not include default_branch_image in location' do
+          location = report.findings.first.location
+
+          expect(location).to be_a(::Gitlab::Ci::Reports::Security::Locations::ContainerScanning)
+          expect(location).to have_attributes(
+            default_branch_image: nil,
+            improved_container_scan_matching_enabled?: true
+          )
+        end
+      end
+
+      context 'when not on default branch' do
+        let(:current_branch) { 'not-default' }
+
+        it 'includes default_branch_image in location' do
+          location = report.findings.first.location
+
+          expect(location).to be_a(::Gitlab::Ci::Reports::Security::Locations::ContainerScanning)
+          expect(location).to have_attributes(
+            default_branch_image: default_branch_image,
+            improved_container_scan_matching_enabled?: true
+          )
+        end
+      end
+    end
+  end
 end

ee/spec/lib/gitlab/ci/reports/security/locations/container_scanning_spec.rb

@@ -18,44 +18,107 @@ RSpec.describe Gitlab::Ci::Reports::Security::Locations::ContainerScanning do
 
   it_behaves_like 'vulnerability location'
 
+  subject { described_class.new(**params) }
+
   describe 'fingerprint' do
     sha1_of = -> (input) { Digest::SHA1.hexdigest(input) }
 
-    subject { described_class.new(**params) }
+    context 'with feature enabled' do
+      where(:image, :default_branch_image, :expected_fingerprint_input) do
+        [
+          ['alpine:3.7.3', nil, 'alpine:3.7.3:glibc'],
+          ['alpine:3.7', nil, 'alpine:3.7:glibc'],
+          ['alpine:8101518288111119448185914762536722131810', nil, 'alpine:glibc'],
+          ['alpine:1.0.0-beta', nil, 'alpine:1.0.0-beta:glibc'],
+          [
+            'gdk.local:5000/group/project/branch:307e0a35643f63652a713d0820db7c388012f724',
+            nil,
+            'gdk.local:5000/group/project/branch:glibc'
+          ],
+          [
+            'registry.gitlab.com/group/project/tmp:af864bd61230d3d694eb01d6205b268b4ad63ac0',
+            nil,
+            'registry.gitlab.com/group/project/tmp:glibc'
+          ],
+          [
+            'registry.gitlab.com/group/project/feature:5b1a4a921d7a50c3757aae3f7df2221878775af4',
+            'registry.gitlab.com/group/project/master:ec301f43f14a2b477806875e49cfc4d3fa0d22c3',
+            'registry.gitlab.com/group/project/master:glibc'
+          ],
+          [
+            'registry.gitlab.com/group/project/feature:d6704dc0b8e33fb550a86f7847d6a3036d4f8bd5',
+            'registry.gitlab.com/group/project:latest',
+            'registry.gitlab.com/group/project:glibc'
+          ],
+          [
+            'registry.gitlab.com/group/project@sha256:a418bbb80b9411f9a08025baa4681e192aaafd16505039bdcb113ccdb90a88fd',
+            'registry.gitlab.com/group/project:latest',
+            'registry.gitlab.com/group/project:glibc'
+          ],
+          [
+            'registry.gitlab.com/group/project/feature:latest',
+            'registry.gitlab.com/group/project:1.0.0',
+            'registry.gitlab.com/group/project:1.0.0:glibc'
+          ]
+        ]
+      end
 
-    specify do
-      params[:image] = 'alpine:3.7.3'
-      expect(subject.fingerprint).to eq(sha1_of.call('alpine:3.7.3:glibc'))
-    end
+      with_them do
+        let(:params) do
+          {
+            image: image,
+            default_branch_image: default_branch_image,
+            operating_system: 'debian:9',
+            package_name: 'glibc',
+            package_version: '1.2.3',
+            improved_container_scan_matching_enabled: true
+          }
+        end
 
-    specify do
-      params[:image] = 'alpine:3.7'
-      expect(subject.fingerprint).to eq(sha1_of.call('alpine:3.7:glibc'))
+        specify { expect(subject.fingerprint).to eq(sha1_of.call(expected_fingerprint_input)) }
+      end
     end
 
-    specify do
-      params[:image] = 'alpine:8101518288111119448185914762536722131810'
-      expect(subject.fingerprint).to eq(sha1_of.call('alpine:glibc'))
-    end
+    context 'with feature disabled' do
+      let(:params) do
+        {
+          image: 'registry.gitlab.com/group/project/feature:ec301f43f14a2b477806875e49cfc4d3fa0d22c3',
+          default_branch_image: 'registry.gitlab.com/group/project/master:ec301f43f14a2b477806875e49cfc4d3fa0d22c3',
+          operating_system: 'debian:9',
+          package_name: 'glibc',
+          package_version: '1.2.3'
+        }
+      end
 
-    specify do
-      params[:image] = 'alpine:1.0.0-beta'
-      expect(subject.fingerprint).to eq(sha1_of.call('alpine:1.0.0-beta:glibc'))
-    end
+      it 'ignores default_branch_image' do
+        expect(subject.fingerprint).to eq(sha1_of.call('registry.gitlab.com/group/project/feature:glibc'))
+      end
 
-    specify do
-      params[:image] = 'registry.gitlab.com/gitlab-org/security-products/analyzers/container-scanning/tmp:af864bd61230d3d694eb01d6205b268b4ad63ac0'
-      expect(subject.fingerprint).to eq(sha1_of.call('registry.gitlab.com/gitlab-org/security-products/analyzers/container-scanning/tmp:glibc'))
-    end
+      where(:image, :expected_fingerprint_input) do
+        [
+          ['alpine:3.7.3', 'alpine:3.7.3:glibc'],
+          ['alpine:3.7', 'alpine:3.7:glibc'],
+          ['alpine:8101518288111119448185914762536722131810', 'alpine:glibc'],
+          ['alpine:1.0.0-beta', 'alpine:1.0.0-beta:glibc'],
+          [
+            'registry.gitlab.com/group/project/tmp:af864bd61230d3d694eb01d6205b268b4ad63ac0',
+            'registry.gitlab.com/group/project/tmp:glibc'
+          ]
+        ]
+      end
 
-    specify do
-      params[:image] = 'registry.gitlab.com/gitlab-org/security-products/tests/container-scanning/master:ec301f43f14a2b477806875e49cfc4d3fa0d22c3'
-      expect(subject.fingerprint).to eq(sha1_of.call('registry.gitlab.com/gitlab-org/security-products/tests/container-scanning/master:glibc'))
-    end
+      with_them do
+        let(:params) do
+          {
+            image: image,
+            operating_system: 'debian:9',
+            package_name: 'glibc',
+            package_version: '1.2.3'
+          }
+        end
 
-    specify do
-      params[:image] = 'registry.gitlab.com/gitlab-org/security-products/dast/webgoat-8.0@sha256:bc09fe2e0721dfaeee79364115aeedf2174cce0947b9ae5fe7c33312ee019a4e'
-      expect(subject.fingerprint).to eq(sha1_of.call('registry.gitlab.com/gitlab-org/security-products/dast/webgoat-8.0@sha256:glibc'))
+        specify { expect(subject.fingerprint).to eq(sha1_of.call(expected_fingerprint_input)) }
+      end
     end
   end
 end