Merge branch 'qa-smoke-tests' into 'master'

add initial smoke tests and documentation

See merge request gitlab-org/gitlab-ce!20568

doc/development/testing_guide/smoke.md

+# Smoke Tests
+
+It is imperative in any testing suite that we have Smoke Tests.  In short, smoke tests are will run quick sanity
+end-to-end functional tests from GitLab QA and are designed to run against the specified environment to ensure that 
+basic functionality is working.
+
+Currently, our suite consists of this basic functionality coverage:
+
+- User Login (Standard Auth)
+- Project Creation
+- Issue Creation
+- Merge Request Creation
+
+---
+
+[Return to Testing documentation](index.md)

doc/development/testing_guide/testing_levels.md

@@ -120,6 +120,14 @@ running feature tests (i.e. using Capybara) against it.
 The actual test scenarios and steps are [part of GitLab Rails] so that they're
 always in-sync with the codebase.
 
+### Smoke tests
+
+Smoke tests are quick tests that may be run at any time (especially after the pre-deployment migrations).
+
+Much like feature tests - these tests run against the UI and ensure that basic functionality is working.
+
+> See [Smoke Tests](smoke.md) for more information.
+
 Read a separate document about [end-to-end tests](end_to_end_tests.md) to
 learn more.
 

qa/README.md

@@ -35,7 +35,7 @@ following call would login to a local [GDK] instance and run all specs in
 `qa/specs/features`:
 
 ```
-bin/qa Test::Instance http://localhost:3000
+bin/qa Test::Instance::All http://localhost:3000
 ```
 
 ### Writing tests
@@ -48,14 +48,14 @@ You can also supply specific tests to run as another parameter. For example, to
 run the repository-related specs, you can execute:
 
 ```
-bin/qa Test::Instance http://localhost qa/specs/features/repository/
+bin/qa Test::Instance::All http://localhost qa/specs/features/repository/
 ```
 
 Since the arguments would be passed to `rspec`, you could use all `rspec`
 options there. For example, passing `--backtrace` and also line number:
 
 ```
-bin/qa Test::Instance http://localhost qa/specs/features/project/create_spec.rb:3 --backtrace
+bin/qa Test::Instance::All http://localhost qa/specs/features/project/create_spec.rb:3 --backtrace
 ```
 
 ### Overriding the authenticated user
@@ -67,7 +67,7 @@ If you need to authenticate as a different user, you can provide the
 `GITLAB_USERNAME` and `GITLAB_PASSWORD` environment variables:
 
 ```
-GITLAB_USERNAME=jsmith GITLAB_PASSWORD=password bin/qa Test::Instance https://gitlab.example.com
+GITLAB_USERNAME=jsmith GITLAB_PASSWORD=password bin/qa Test::Instance::All https://gitlab.example.com
 ```
 
 If your user doesn't have permission to default sandbox group
@@ -75,13 +75,13 @@ If your user doesn't have permission to default sandbox group
 `GITLAB_SANDBOX_NAME`:
 
 ```
-GITLAB_USERNAME=jsmith GITLAB_PASSWORD=password GITLAB_SANDBOX_NAME=jsmith-qa-sandbox bin/qa Test::Instance https://gitlab.example.com
+GITLAB_USERNAME=jsmith GITLAB_PASSWORD=password GITLAB_SANDBOX_NAME=jsmith-qa-sandbox bin/qa Test::Instance::All https://gitlab.example.com
 ```
 
 In addition, the `GITLAB_USER_TYPE` can be set to "ldap" to sign in as an LDAP user:
 
 ```
-GITLAB_USER_TYPE=ldap GITLAB_USERNAME=jsmith GITLAB_PASSWORD=password GITLAB_SANDBOX_NAME=jsmith-qa-sandbox bin/qa Test::Instance https://gitlab.example.com
+GITLAB_USER_TYPE=ldap GITLAB_USERNAME=jsmith GITLAB_PASSWORD=password GITLAB_SANDBOX_NAME=jsmith-qa-sandbox bin/qa Test::Instance::All https://gitlab.example.com
 ```
 
 All [supported environment variables are here](https://gitlab.com/gitlab-org/gitlab-qa#supported-environment-variables).

qa/qa.rb

@@ -77,14 +77,16 @@ module QA
     #
     autoload :Bootable, 'qa/scenario/bootable'
     autoload :Actable, 'qa/scenario/actable'
-    autoload :Taggable, 'qa/scenario/taggable'
     autoload :Template, 'qa/scenario/template'
 
     ##
     # Test scenario entrypoints.
     #
     module Test
-      autoload :Instance, 'qa/scenario/test/instance'
+      module Instance
+        autoload :All, 'qa/scenario/test/instance/all'
+        autoload :Smoke, 'qa/scenario/test/instance/smoke'
+      end
 
       module Integration
         autoload :Github, 'qa/scenario/test/integration/github'

qa/qa/scenario/taggable.rb

-module QA
-  module Scenario
-    module Taggable
-      # rubocop:disable Gitlab/ModuleWithInstanceVariables
-
-      def tags(*tags)
-        @tags = tags
-      end
-
-      def focus
-        @tags.to_a
-      end
-
-      # rubocop:enable Gitlab/ModuleWithInstanceVariables
-    end
-  end
-end

qa/qa/scenario/template.rb

 module QA
   module Scenario
     class Template
-      def self.perform(*args)
+      class << self
+        def perform(*args)
           new.tap do |scenario|
             yield scenario if block_given?
             break scenario.perform(*args)
           end
         end
 
-      def perform(*_args)
-        raise NotImplementedError
+        def tags(*tags)
+          @tags = tags
+        end
+
+        def focus
+          @tags.to_a
+        end
+      end
+
+      def perform(address, *rspec_options)
+        Runtime::Scenario.define(:gitlab_address, address)
+
+        Specs::Runner.perform do |specs|
+          specs.tty = true
+          specs.tags = self.class.focus
+          specs.options =
+            if rspec_options.any?
+              rspec_options
+            else
+              File.expand_path('../../specs/features', __dir__)
+            end
+        end
       end
     end
   end

qa/qa/scenario/test/instance.rb

-module QA
-  module Scenario
-    module Test
-      ##
-      # Base class for running the suite against any GitLab instance,
-      # including staging and on-premises installation.
-      #
-      class Instance < Template
-        include Bootable
-        extend Taggable
-
-        tags :core
-
-        def perform(address, *rspec_options)
-          Runtime::Scenario.define(:gitlab_address, address)
-
-          ##
-          # Perform before hooks, which are different for CE and EE
-          #
-          Runtime::Release.perform_before_hooks
-
-          Specs::Runner.perform do |specs|
-            specs.tty = true
-            specs.tags = self.class.focus
-            specs.options =
-              if rspec_options.any?
-                rspec_options
-              else
-                ::File.expand_path('../../specs/features', __dir__)
-              end
-          end
-        end
-      end
-    end
-  end
-end

qa/qa/scenario/test/instance/all.rb

+module QA
+  module Scenario
+    module Test
+      ##
+      # Base class for running the suite against any GitLab instance,
+      # including staging and on-premises installation.
+      #
+      module Instance
+        class All < Template
+          include Bootable
+        end
+      end
+    end
+  end
+end

qa/qa/scenario/test/instance/smoke.rb

+module QA
+  module Scenario
+    module Test
+      module Instance
+        ##
+        # Base class for running the suite against any GitLab instance,
+        # including staging and on-premises installation.
+        #
+        class Smoke < Template
+          include Bootable
+
+          tags :smoke
+        end
+      end
+    end
+  end
+end

qa/qa/specs/features/api/basics_spec.rb

 require 'securerandom'
 
 module QA
-  describe 'API basics', :core do
+  describe 'API basics' do
     before(:context) do
       @api_client = Runtime::API::Client.new(:gitlab)
     end

qa/qa/specs/features/api/users_spec.rb

 module QA
-  describe 'API users', :core do
+  describe 'API users' do
     before(:context) do
       @api_client = Runtime::API::Client.new(:gitlab)
     end

qa/qa/specs/features/login/basic_spec.rb

+module QA
+  describe 'basic user login', :smoke do
+    it 'user logs in using basic credentials' do
+      Runtime::Browser.visit(:gitlab, Page::Main::Login)
+      Page::Main::Login.act { sign_in_using_credentials }
+
+      # TODO, since `Signed in successfully` message was removed
+      # this is the only way to tell if user is signed in correctly.
+      #
+      Page::Menu::Main.perform do |menu|
+        expect(menu).to have_personal_area
+      end
+    end
+  end
+end

qa/qa/specs/features/login/ldap_spec.rb

 module QA
-  describe 'LDAP user login', :ldap do
+  describe 'LDAP user login', :orchestrated, :ldap do
     before do
       Runtime::Env.user_type = 'ldap'
     end

qa/qa/specs/features/mattermost/group_create_spec.rb

 module QA
-  describe 'create a new group', :mattermost do
+  describe 'create a new group', :orchestrated, :mattermost do
     it 'creating a group with a mattermost team' do
       Runtime::Browser.visit(:gitlab, Page::Main::Login)
       Page::Main::Login.act { sign_in_using_credentials }

qa/qa/specs/features/mattermost/login_spec.rb

 module QA
-  describe 'logging in to Mattermost', :mattermost do
+  describe 'logging in to Mattermost', :orchestrated, :mattermost do
     it 'can use gitlab oauth' do
       Runtime::Browser.visit(:gitlab, Page::Main::Login) do
         Page::Main::Login.act { sign_in_using_credentials }

qa/qa/specs/features/merge_request/create_spec.rb

 module QA
-  describe 'creates a merge request', :core do
+  describe 'creates a merge request with milestone' do
     it 'user creates a new merge request'  do
       Runtime::Browser.visit(:gitlab, Page::Main::Login)
       Page::Main::Login.act { sign_in_using_credentials }
@@ -29,4 +29,25 @@ module QA
       end
     end
   end
+
+  describe 'creates a merge request', :smoke do
+    it 'user creates a new merge request'  do
+      Runtime::Browser.visit(:gitlab, Page::Main::Login)
+      Page::Main::Login.act { sign_in_using_credentials }
+
+      current_project = Factory::Resource::Project.fabricate! do |project|
+        project.name = 'project-with-merge-request'
+      end
+
+      Factory::Resource::MergeRequest.fabricate! do |merge_request|
+        merge_request.title = 'This is a merge request'
+        merge_request.description = 'Great feature'
+        merge_request.project = current_project
+      end
+
+      expect(page).to have_content('This is a merge request')
+      expect(page).to have_content('Great feature')
+      expect(page).to have_content(/Opened [\w\s]+ ago/)
+    end
+  end
 end

qa/qa/specs/features/merge_request/rebase_spec.rb

 module QA
-  describe 'merge request rebase', :core do
+  describe 'merge request rebase' do
     it 'rebases source branch of merge request'  do
       Runtime::Browser.visit(:gitlab, Page::Main::Login)
       Page::Main::Login.act { sign_in_using_credentials }

qa/qa/specs/features/merge_request/squash_spec.rb

 module QA
-  describe 'merge request squash commits', :core do
+  describe 'merge request squash commits' do
     it 'when squash commits is marked before merge'  do
       Runtime::Browser.visit(:gitlab, Page::Main::Login)
       Page::Main::Login.act { sign_in_using_credentials }

qa/qa/specs/features/project/activity_spec.rb

 module QA
-  describe 'activity page', :core do
+  describe 'activity page' do
     it 'push creates an event in the activity page' do
       Runtime::Browser.visit(:gitlab, Page::Main::Login)
       Page::Main::Login.act { sign_in_using_credentials }

qa/qa/specs/features/project/add_deploy_key_spec.rb

 module QA
-  describe 'deploy keys support', :core do
+  describe 'deploy keys support' do
     it 'user adds a deploy key' do
       Runtime::Browser.visit(:gitlab, Page::Main::Login)
       Page::Main::Login.act { sign_in_using_credentials }

qa/qa/specs/features/project/add_secret_variable_spec.rb

 module QA
-  describe 'secret variables support', :core do
+  describe 'secret variables support' do
     it 'user adds a secret variable' do
       Runtime::Browser.visit(:gitlab, Page::Main::Login)
       Page::Main::Login.act { sign_in_using_credentials }

qa/qa/specs/features/project/auto_devops_spec.rb

 require 'pathname'
 
 module QA
-  describe 'Auto Devops', :kubernetes do
+  describe 'Auto Devops', :orchestrated, :kubernetes do
     after do
       @cluster&.remove!
     end

qa/qa/specs/features/project/create_issue_spec.rb

 module QA
-  describe 'creates issue', :core do
+  describe 'creates issue', :smoke do
     let(:issue_title) { 'issue title' }
 
     it 'user creates issue' do

qa/qa/specs/features/project/create_spec.rb

 module QA
-  describe 'create a new project', :core do
+  describe 'create a new project', :smoke do
     it 'user creates a new project' do
       Runtime::Browser.visit(:gitlab, Page::Main::Login)
       Page::Main::Login.act { sign_in_using_credentials }

qa/qa/specs/features/project/deploy_key_clone_spec.rb

 require 'digest/sha1'
 
 module QA
-  describe 'cloning code using a deploy key', :core, :docker do
+  describe 'cloning code using a deploy key', :docker do
     def login
       Runtime::Browser.visit(:gitlab, Page::Main::Login)
       Page::Main::Login.act { sign_in_using_credentials }

qa/qa/specs/features/project/fork_project_spec.rb

 module QA
-  describe 'Project fork', :core do
+  describe 'Project fork' do
     it 'can submit merge requests to upstream master' do
       Runtime::Browser.visit(:gitlab, Page::Main::Login)
       Page::Main::Login.act { sign_in_using_credentials }

qa/qa/specs/features/project/import_from_github_spec.rb

 module QA
-  describe 'user imports a GitHub repo', :core, :github do
+  describe 'user imports a GitHub repo', :orchestrated, :github do
     let(:imported_project) do
       Factory::Resource::ProjectImportedFromGithub.fabricate! do |project|
         project.name = 'imported-project'

qa/qa/specs/features/project/pipelines_spec.rb

 module QA
-  describe 'CI/CD Pipelines', :core, :docker do
+  describe 'CI/CD Pipelines', :orchestrated, :docker do
     let(:executor) { "qa-runner-#{Time.now.to_i}" }
 
     after do

qa/qa/specs/features/project/wikis_spec.rb

 module QA
-  describe 'Wiki Functionality', :core do
+  describe 'Wiki Functionality' do
     def login
       Runtime::Browser.visit(:gitlab, Page::Main::Login)
       Page::Main::Login.act { sign_in_using_credentials }

qa/qa/specs/features/repository/clone_spec.rb

 module QA
-  describe 'clone code from the repository', :core do
+  describe 'clone code from the repository' do
     context 'with regular account over http' do
       let(:location) do
         Page::Project::Show.act do

qa/qa/specs/features/repository/protected_branches_spec.rb

 module QA
-  describe 'branch protection support', :core do
+  describe 'branch protection support' do
     let(:branch_name) { 'protected-branch' }
     let(:commit_message) { 'Protected push commit message' }
     let(:project) do

qa/qa/specs/features/repository/push_spec.rb

 module QA
-  describe 'push code to repository', :core do
+  describe 'push code to repository' do
     context 'with regular account over http' do
       it 'user pushes code to the repository'  do
         Runtime::Browser.visit(:gitlab, Page::Main::Login)

qa/qa/specs/runner.rb

@@ -14,7 +14,13 @@ module QA
       def perform
         args = []
         args.push('--tty') if tty
-        tags.to_a.each { |tag| args.push(['-t', tag.to_s]) }
+
+        if tags.any?
+          tags.each { |tag| args.push(['-t', tag.to_s]) }
+        else
+          args.push(%w[-t ~orchestrated])
+        end
+
         args.push(options)
 
         Runtime::Browser.configure!

qa/spec/scenario/test/instance_spec.rb → qa/spec/scenario/test/instance/all_spec.rb

-describe QA::Scenario::Test::Instance do
-  subject do
-    Class.new(described_class) do
-      tags :rspec
-    end
-  end
-
+describe QA::Scenario::Test::Instance::All do
   context '#perform' do
     let(:arguments) { spy('Runtime::Scenario') }
     let(:release) { spy('Runtime::Release') }
@@ -26,16 +20,16 @@ describe QA::Scenario::Test::Instance do
     end
 
     context 'no paths' do
-      it 'should call runner with default arguments' do
+      it 'calls runner with default arguments' do
         subject.perform("test")
 
         expect(runner).to have_received(:options=)
-          .with(::File.expand_path('../../../qa/specs/features', __dir__))
+          .with(::File.expand_path('../../../../../qa/specs/features', __dir__))
       end
     end
 
     context 'specifying paths' do
-      it 'should call runner with paths' do
+      it 'calls runner with paths' do
         subject.perform('test', 'path1', 'path2')
 
         expect(runner).to have_received(:options=).with(%w[path1 path2])

qa/spec/scenario/test/instance/smoke_spec.rb

+describe QA::Scenario::Test::Instance::Smoke do
+  subject { Class.new(described_class) { tags :smoke } }
+
+  context '#perform' do
+    let(:arguments) { spy('Runtime::Scenario') }
+    let(:release) { spy('Runtime::Release') }
+    let(:runner) { spy('Specs::Runner') }
+
+    before do
+      stub_const('QA::Runtime::Release', release)
+      stub_const('QA::Runtime::Scenario', arguments)
+      stub_const('QA::Specs::Runner', runner)
+
+      allow(runner).to receive(:perform).and_yield(runner)
+    end
+
+    it 'sets an address of the subject' do
+      subject.perform("hello")
+
+      expect(arguments).to have_received(:define)
+        .with(:gitlab_address, "hello")
+    end
+
+    it 'has a smoke tag' do
+      expect(subject.focus).to eq([:smoke]) # rubocop:disable Focus
+    end
+
+    context 'no paths' do
+      it 'calls runner with default arguments' do
+        subject.perform("test")
+
+        expect(runner).to have_received(:options=)
+          .with(File.expand_path('../../../../../qa/specs/features', __dir__))
+      end
+    end
+
+    context 'specifying paths' do
+      it 'calls runner with paths' do
+        subject.perform('test', 'path1', 'path2')
+
+        expect(runner).to have_received(:options=).with(%w[path1 path2])
+      end
+    end
+  end
+end