Commit 87c491b7 authored by Rémy Coutable's avatar Rémy Coutable

Merge branch 'qa-smoke-tests' into 'master'

add initial smoke tests and documentation

See merge request gitlab-org/gitlab-ce!20568
parents c5bf09e7 d67936b6
# Smoke Tests
It is imperative in any testing suite that we have Smoke Tests. In short, smoke tests are will run quick sanity
end-to-end functional tests from GitLab QA and are designed to run against the specified environment to ensure that
basic functionality is working.
Currently, our suite consists of this basic functionality coverage:
- User Login (Standard Auth)
- Project Creation
- Issue Creation
- Merge Request Creation
---
[Return to Testing documentation](index.md)
......@@ -120,6 +120,14 @@ running feature tests (i.e. using Capybara) against it.
The actual test scenarios and steps are [part of GitLab Rails] so that they're
always in-sync with the codebase.
### Smoke tests
Smoke tests are quick tests that may be run at any time (especially after the pre-deployment migrations).
Much like feature tests - these tests run against the UI and ensure that basic functionality is working.
> See [Smoke Tests](smoke.md) for more information.
Read a separate document about [end-to-end tests](end_to_end_tests.md) to
learn more.
......
......@@ -35,7 +35,7 @@ following call would login to a local [GDK] instance and run all specs in
`qa/specs/features`:
```
bin/qa Test::Instance http://localhost:3000
bin/qa Test::Instance::All http://localhost:3000
```
### Writing tests
......@@ -48,14 +48,14 @@ You can also supply specific tests to run as another parameter. For example, to
run the repository-related specs, you can execute:
```
bin/qa Test::Instance http://localhost qa/specs/features/repository/
bin/qa Test::Instance::All http://localhost qa/specs/features/repository/
```
Since the arguments would be passed to `rspec`, you could use all `rspec`
options there. For example, passing `--backtrace` and also line number:
```
bin/qa Test::Instance http://localhost qa/specs/features/project/create_spec.rb:3 --backtrace
bin/qa Test::Instance::All http://localhost qa/specs/features/project/create_spec.rb:3 --backtrace
```
### Overriding the authenticated user
......@@ -67,7 +67,7 @@ If you need to authenticate as a different user, you can provide the
`GITLAB_USERNAME` and `GITLAB_PASSWORD` environment variables:
```
GITLAB_USERNAME=jsmith GITLAB_PASSWORD=password bin/qa Test::Instance https://gitlab.example.com
GITLAB_USERNAME=jsmith GITLAB_PASSWORD=password bin/qa Test::Instance::All https://gitlab.example.com
```
If your user doesn't have permission to default sandbox group
......@@ -75,13 +75,13 @@ If your user doesn't have permission to default sandbox group
`GITLAB_SANDBOX_NAME`:
```
GITLAB_USERNAME=jsmith GITLAB_PASSWORD=password GITLAB_SANDBOX_NAME=jsmith-qa-sandbox bin/qa Test::Instance https://gitlab.example.com
GITLAB_USERNAME=jsmith GITLAB_PASSWORD=password GITLAB_SANDBOX_NAME=jsmith-qa-sandbox bin/qa Test::Instance::All https://gitlab.example.com
```
In addition, the `GITLAB_USER_TYPE` can be set to "ldap" to sign in as an LDAP user:
```
GITLAB_USER_TYPE=ldap GITLAB_USERNAME=jsmith GITLAB_PASSWORD=password GITLAB_SANDBOX_NAME=jsmith-qa-sandbox bin/qa Test::Instance https://gitlab.example.com
GITLAB_USER_TYPE=ldap GITLAB_USERNAME=jsmith GITLAB_PASSWORD=password GITLAB_SANDBOX_NAME=jsmith-qa-sandbox bin/qa Test::Instance::All https://gitlab.example.com
```
All [supported environment variables are here](https://gitlab.com/gitlab-org/gitlab-qa#supported-environment-variables).
......
......@@ -77,14 +77,16 @@ module QA
#
autoload :Bootable, 'qa/scenario/bootable'
autoload :Actable, 'qa/scenario/actable'
autoload :Taggable, 'qa/scenario/taggable'
autoload :Template, 'qa/scenario/template'
##
# Test scenario entrypoints.
#
module Test
autoload :Instance, 'qa/scenario/test/instance'
module Instance
autoload :All, 'qa/scenario/test/instance/all'
autoload :Smoke, 'qa/scenario/test/instance/smoke'
end
module Integration
autoload :Github, 'qa/scenario/test/integration/github'
......
module QA
module Scenario
module Taggable
# rubocop:disable Gitlab/ModuleWithInstanceVariables
def tags(*tags)
@tags = tags
end
def focus
@tags.to_a
end
# rubocop:enable Gitlab/ModuleWithInstanceVariables
end
end
end
module QA
module Scenario
class Template
def self.perform(*args)
new.tap do |scenario|
yield scenario if block_given?
break scenario.perform(*args)
class << self
def perform(*args)
new.tap do |scenario|
yield scenario if block_given?
break scenario.perform(*args)
end
end
def tags(*tags)
@tags = tags
end
def focus
@tags.to_a
end
end
def perform(*_args)
raise NotImplementedError
def perform(address, *rspec_options)
Runtime::Scenario.define(:gitlab_address, address)
Specs::Runner.perform do |specs|
specs.tty = true
specs.tags = self.class.focus
specs.options =
if rspec_options.any?
rspec_options
else
File.expand_path('../../specs/features', __dir__)
end
end
end
end
end
......
module QA
module Scenario
module Test
##
# Base class for running the suite against any GitLab instance,
# including staging and on-premises installation.
#
class Instance < Template
include Bootable
extend Taggable
tags :core
def perform(address, *rspec_options)
Runtime::Scenario.define(:gitlab_address, address)
##
# Perform before hooks, which are different for CE and EE
#
Runtime::Release.perform_before_hooks
Specs::Runner.perform do |specs|
specs.tty = true
specs.tags = self.class.focus
specs.options =
if rspec_options.any?
rspec_options
else
::File.expand_path('../../specs/features', __dir__)
end
end
end
end
end
end
end
module QA
module Scenario
module Test
##
# Base class for running the suite against any GitLab instance,
# including staging and on-premises installation.
#
module Instance
class All < Template
include Bootable
end
end
end
end
end
module QA
module Scenario
module Test
module Instance
##
# Base class for running the suite against any GitLab instance,
# including staging and on-premises installation.
#
class Smoke < Template
include Bootable
tags :smoke
end
end
end
end
end
require 'securerandom'
module QA
describe 'API basics', :core do
describe 'API basics' do
before(:context) do
@api_client = Runtime::API::Client.new(:gitlab)
end
......
module QA
describe 'API users', :core do
describe 'API users' do
before(:context) do
@api_client = Runtime::API::Client.new(:gitlab)
end
......
module QA
describe 'basic user login', :smoke do
it 'user logs in using basic credentials' do
Runtime::Browser.visit(:gitlab, Page::Main::Login)
Page::Main::Login.act { sign_in_using_credentials }
# TODO, since `Signed in successfully` message was removed
# this is the only way to tell if user is signed in correctly.
#
Page::Menu::Main.perform do |menu|
expect(menu).to have_personal_area
end
end
end
end
module QA
describe 'LDAP user login', :ldap do
describe 'LDAP user login', :orchestrated, :ldap do
before do
Runtime::Env.user_type = 'ldap'
end
......
module QA
describe 'create a new group', :mattermost do
describe 'create a new group', :orchestrated, :mattermost do
it 'creating a group with a mattermost team' do
Runtime::Browser.visit(:gitlab, Page::Main::Login)
Page::Main::Login.act { sign_in_using_credentials }
......
module QA
describe 'logging in to Mattermost', :mattermost do
describe 'logging in to Mattermost', :orchestrated, :mattermost do
it 'can use gitlab oauth' do
Runtime::Browser.visit(:gitlab, Page::Main::Login) do
Page::Main::Login.act { sign_in_using_credentials }
......
module QA
describe 'creates a merge request', :core do
describe 'creates a merge request with milestone' do
it 'user creates a new merge request' do
Runtime::Browser.visit(:gitlab, Page::Main::Login)
Page::Main::Login.act { sign_in_using_credentials }
......@@ -29,4 +29,25 @@ module QA
end
end
end
describe 'creates a merge request', :smoke do
it 'user creates a new merge request' do
Runtime::Browser.visit(:gitlab, Page::Main::Login)
Page::Main::Login.act { sign_in_using_credentials }
current_project = Factory::Resource::Project.fabricate! do |project|
project.name = 'project-with-merge-request'
end
Factory::Resource::MergeRequest.fabricate! do |merge_request|
merge_request.title = 'This is a merge request'
merge_request.description = 'Great feature'
merge_request.project = current_project
end
expect(page).to have_content('This is a merge request')
expect(page).to have_content('Great feature')
expect(page).to have_content(/Opened [\w\s]+ ago/)
end
end
end
module QA
describe 'merge request rebase', :core do
describe 'merge request rebase' do
it 'rebases source branch of merge request' do
Runtime::Browser.visit(:gitlab, Page::Main::Login)
Page::Main::Login.act { sign_in_using_credentials }
......
module QA
describe 'merge request squash commits', :core do
describe 'merge request squash commits' do
it 'when squash commits is marked before merge' do
Runtime::Browser.visit(:gitlab, Page::Main::Login)
Page::Main::Login.act { sign_in_using_credentials }
......
module QA
describe 'activity page', :core do
describe 'activity page' do
it 'push creates an event in the activity page' do
Runtime::Browser.visit(:gitlab, Page::Main::Login)
Page::Main::Login.act { sign_in_using_credentials }
......
module QA
describe 'deploy keys support', :core do
describe 'deploy keys support' do
it 'user adds a deploy key' do
Runtime::Browser.visit(:gitlab, Page::Main::Login)
Page::Main::Login.act { sign_in_using_credentials }
......
module QA
describe 'secret variables support', :core do
describe 'secret variables support' do
it 'user adds a secret variable' do
Runtime::Browser.visit(:gitlab, Page::Main::Login)
Page::Main::Login.act { sign_in_using_credentials }
......
require 'pathname'
module QA
describe 'Auto Devops', :kubernetes do
describe 'Auto Devops', :orchestrated, :kubernetes do
after do
@cluster&.remove!
end
......
module QA
describe 'creates issue', :core do
describe 'creates issue', :smoke do
let(:issue_title) { 'issue title' }
it 'user creates issue' do
......
module QA
describe 'create a new project', :core do
describe 'create a new project', :smoke do
it 'user creates a new project' do
Runtime::Browser.visit(:gitlab, Page::Main::Login)
Page::Main::Login.act { sign_in_using_credentials }
......
require 'digest/sha1'
module QA
describe 'cloning code using a deploy key', :core, :docker do
describe 'cloning code using a deploy key', :docker do
def login
Runtime::Browser.visit(:gitlab, Page::Main::Login)
Page::Main::Login.act { sign_in_using_credentials }
......
module QA
describe 'Project fork', :core do
describe 'Project fork' do
it 'can submit merge requests to upstream master' do
Runtime::Browser.visit(:gitlab, Page::Main::Login)
Page::Main::Login.act { sign_in_using_credentials }
......
module QA
describe 'user imports a GitHub repo', :core, :github do
describe 'user imports a GitHub repo', :orchestrated, :github do
let(:imported_project) do
Factory::Resource::ProjectImportedFromGithub.fabricate! do |project|
project.name = 'imported-project'
......
module QA
describe 'CI/CD Pipelines', :core, :docker do
describe 'CI/CD Pipelines', :orchestrated, :docker do
let(:executor) { "qa-runner-#{Time.now.to_i}" }
after do
......
module QA
describe 'Wiki Functionality', :core do
describe 'Wiki Functionality' do
def login
Runtime::Browser.visit(:gitlab, Page::Main::Login)
Page::Main::Login.act { sign_in_using_credentials }
......
module QA
describe 'clone code from the repository', :core do
describe 'clone code from the repository' do
context 'with regular account over http' do
let(:location) do
Page::Project::Show.act do
......
module QA
describe 'branch protection support', :core do
describe 'branch protection support' do
let(:branch_name) { 'protected-branch' }
let(:commit_message) { 'Protected push commit message' }
let(:project) do
......
module QA
describe 'push code to repository', :core do
describe 'push code to repository' do
context 'with regular account over http' do
it 'user pushes code to the repository' do
Runtime::Browser.visit(:gitlab, Page::Main::Login)
......
......@@ -14,7 +14,13 @@ module QA
def perform
args = []
args.push('--tty') if tty
tags.to_a.each { |tag| args.push(['-t', tag.to_s]) }
if tags.any?
tags.each { |tag| args.push(['-t', tag.to_s]) }
else
args.push(%w[-t ~orchestrated])
end
args.push(options)
Runtime::Browser.configure!
......
describe QA::Scenario::Test::Instance do
subject do
Class.new(described_class) do
tags :rspec
end
end
describe QA::Scenario::Test::Instance::All do
context '#perform' do
let(:arguments) { spy('Runtime::Scenario') }
let(:release) { spy('Runtime::Release') }
......@@ -26,16 +20,16 @@ describe QA::Scenario::Test::Instance do
end
context 'no paths' do
it 'should call runner with default arguments' do
it 'calls runner with default arguments' do
subject.perform("test")
expect(runner).to have_received(:options=)
.with(::File.expand_path('../../../qa/specs/features', __dir__))
.with(::File.expand_path('../../../../../qa/specs/features', __dir__))
end
end
context 'specifying paths' do
it 'should call runner with paths' do
it 'calls runner with paths' do
subject.perform('test', 'path1', 'path2')
expect(runner).to have_received(:options=).with(%w[path1 path2])
......
describe QA::Scenario::Test::Instance::Smoke do
subject { Class.new(described_class) { tags :smoke } }
context '#perform' do
let(:arguments) { spy('Runtime::Scenario') }
let(:release) { spy('Runtime::Release') }
let(:runner) { spy('Specs::Runner') }
before do
stub_const('QA::Runtime::Release', release)
stub_const('QA::Runtime::Scenario', arguments)
stub_const('QA::Specs::Runner', runner)
allow(runner).to receive(:perform).and_yield(runner)
end
it 'sets an address of the subject' do
subject.perform("hello")
expect(arguments).to have_received(:define)
.with(:gitlab_address, "hello")
end
it 'has a smoke tag' do
expect(subject.focus).to eq([:smoke]) # rubocop:disable Focus
end
context 'no paths' do
it 'calls runner with default arguments' do
subject.perform("test")
expect(runner).to have_received(:options=)
.with(File.expand_path('../../../../../qa/specs/features', __dir__))
end
end
context 'specifying paths' do
it 'calls runner with paths' do
subject.perform('test', 'path1', 'path2')
expect(runner).to have_received(:options=).with(%w[path1 path2])
end
end
end
end
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment