Commit f9bc8d20 authored by Stan Hu's avatar Stan Hu

Merge branch '27376-go-package-mvc' into 'master'

Go Module Proxy MVC

Closes #27376

See merge request gitlab-org/gitlab!27746
parents a57593b7 d5e63a6d
...@@ -66,6 +66,12 @@ class Settings < Settingslogic ...@@ -66,6 +66,12 @@ class Settings < Settingslogic
(base_url(gitlab) + [gitlab.relative_url_root]).join('') (base_url(gitlab) + [gitlab.relative_url_root]).join('')
end end
def build_gitlab_go_url
# "Go package paths are not URLs, and do not include port numbers"
# https://github.com/golang/go/issues/38213#issuecomment-607851460
"#{gitlab.host}#{gitlab.relative_url_root}"
end
def kerberos_protocol def kerberos_protocol
kerberos.https ? "https" : "http" kerberos.https ? "https" : "http"
end end
......
...@@ -13,6 +13,7 @@ The Packages feature allows GitLab to act as a repository for the following: ...@@ -13,6 +13,7 @@ The Packages feature allows GitLab to act as a repository for the following:
| [Conan Repository](../../user/packages/conan_repository/index.md) | The GitLab Conan Repository enables every project in GitLab to have its own space to store [Conan](https://conan.io/) packages. | 12.4+ | | [Conan Repository](../../user/packages/conan_repository/index.md) | The GitLab Conan Repository enables every project in GitLab to have its own space to store [Conan](https://conan.io/) packages. | 12.4+ |
| [Maven Repository](../../user/packages/maven_repository/index.md) | The GitLab Maven Repository enables every project in GitLab to have its own space to store [Maven](https://maven.apache.org/) packages. | 11.3+ | | [Maven Repository](../../user/packages/maven_repository/index.md) | The GitLab Maven Repository enables every project in GitLab to have its own space to store [Maven](https://maven.apache.org/) packages. | 11.3+ |
| [NPM Registry](../../user/packages/npm_registry/index.md) | The GitLab NPM Registry enables every project in GitLab to have its own space to store [NPM](https://www.npmjs.com/) packages. | 11.7+ | | [NPM Registry](../../user/packages/npm_registry/index.md) | The GitLab NPM Registry enables every project in GitLab to have its own space to store [NPM](https://www.npmjs.com/) packages. | 11.7+ |
| [Go Proxy](../../user/packages/go_proxy/index.md) | The Go proxy for GitLab enables every project in GitLab to be fetched with the [Go proxy protocol](https://proxy.golang.org/). | 13.1+ |
Don't you see your package management system supported yet? Don't you see your package management system supported yet?
Please consider contributing Please consider contributing
......
# GitLab Go Proxy **(PREMIUM)**
> - [Introduced](https://gitlab.com/gitlab-org/gitlab/issues/27376) in [GitLab Premium](https://about.gitlab.com/pricing/) 13.1.
> - It's deployed behind a feature flag, disabled by default.
> - It's disabled on GitLab.com.
> - It's not recommended for production use.
> - To use it in GitLab self-managed instances, ask a GitLab administrator to [enable it](#enable-the-go-proxy). **(PREMIUM)**
With the Go proxy for GitLab, every project in GitLab can be fetched with the
[Go proxy protocol](https://proxy.golang.org/).
## Prerequisites
### Enable the Go proxy
The Go proxy for GitLab is under development and not ready for production use, due to
[potential performance issues with large repositories](https://gitlab.com/gitlab-org/gitlab/-/issues/218083).
It is deployed behind a feature flag that is **disabled by default**.
[GitLab administrators with access to the GitLab Rails console](../../../administration/feature_flags.md)
can enable it for your instance.
To enable it:
```ruby
Feature.enable(:go_proxy) # or
```
To disable it:
```ruby
Feature.disable(:go_proxy)
```
To enable or disable it for specific projects:
```ruby
Feature.enable(:go_proxy, Project.find(1))
Feature.disable(:go_proxy, Project.find(2))
```
### Enable the Package Registry
The Package Registry is enabled for new projects by default. If you cannot find
the **{package}** **Packages > List** entry under your project's sidebar, verify
the following:
1. Your GitLab administrator has
[enabled support for the Package Registry](../../../administration/packages/index.md). **(PREMIUM ONLY)**
1. The Package Registry is [enabled for your project](../index.md).
NOTE: **Note:**
GitLab does not currently display Go modules in the **Packages Registry** of a project.
Follow [this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/213770) for details.
### Fetch modules from private projects
`go` does not support transmitting credentials over insecure connections. The
steps below work only if GitLab is configured for HTTPS.
1. Configure Go to include HTTP basic authentication credentials when fetching
from the Go proxy for GitLab.
1. Configure Go to skip downloading of checksums for private GitLab projects
from the public checksum database.
#### Enable Request Authentication
Create a [personal access token](../../profile/personal_access_tokens.md) with
the `api` or `read_api` scope and add it to
[`~/.netrc`](https://ec.haxx.se/usingcurl/usingcurl-netrc):
```netrc
machine <url> login <username> password <token>
```
`<url>` should be the URL of the GitLab instance, for example `gitlab.com`.
`<username>` and `<token>` should be your username and the personal access
token, respectively.
#### Disable checksum database queries
Go can be configured to query a checksum database for module checksums. Go 1.13
and later query `sum.golang.org` by default. This fails for modules that are not
public and thus not accessible to `sum.golang.org`. To resolve this issue, set
`GONOSUMDB` to a comma-separated list of projects or namespaces for which Go
should not query the checksum database. For example, `go env -w
GONOSUMDB=gitlab.com/my/project` persistently configures Go to skip checksum
queries for the project `gitlab.com/my/project`.
Checksum database queries can be disabled for arbitrary prefixes or disabled
entirely. However, checksum database queries are a security mechanism and as
such they should be disabled selectively and only when necessary. `GOSUMDB=off`
or `GONOSUMDB=*` disables checksum queries entirely. `GONOSUMDB=gitlab.com`
disables checksum queries for all projects hosted on GitLab.com.
## Add GitLab as a Go proxy
NOTE: **Note:**
To use a Go proxy, you must be using Go 1.13 or later.
The available proxy endpoints are:
- Project - can fetch modules defined by a project - `/api/v4/projects/:id/packages/go`
Go's use of proxies is configured with the `GOPROXY` environment variable, as a
comma separated list of URLs. Go 1.14 adds support for comma separated list of
URLs. Go 1.14 adds support for using `go env -w` to manage Go's environment
variables. For example, `go env -w GOPROXY=...` writes to `$GOPATH/env`
(which defaults to `~/.go/env`). `GOPROXY` can also be configured as a normal
environment variable, with RC files or `export GOPROXY=...`.
The default value of `$GOPROXY` is `https://proxy.golang.org,direct`, which
tells `go` to first query `proxy.golang.org` and fallback to direct VCS
operations (`git clone`, `svc checkout`, etc). Replacing
`https://proxy.golang.org` with a GitLab endpoint will direct all fetches
through GitLab. Currently GitLab's Go proxy does not support dependency
proxying, so all external dependencies will be handled directly. If GitLab's
endpoint is inserted before `https://proxy.golang.org`, then all fetches will
first go through GitLab. This can help avoid making requests for private
packages to the public proxy, but `GOPRIVATE` is a much safer way of achieving
that.
For example, with the following configuration, Go will attempt to fetch modules
from 1) GitLab project 1234's Go module proxy, 2) `proxy.golang.org`, and
finally 3) directly with Git (or another VCS, depending on where the module
source is hosted).
```shell
go env -w GOPROXY=https://gitlab.com/api/v4/projects/1234/packages/go,https://proxy.golang.org,direct
```
## Release a module
Go modules and module versions are handled entirely with Git (or SVN, Mercurial,
and so on). A module is a repository containing Go source and a `go.mod` file. A
version of a module is a Git tag (or equivalent) that is a valid [semantic
version](https://semver.org), prefixed with 'v'. For example, `v1.0.0` and
`v1.3.2-alpha` are valid module versions, but `v1` or `v1.2` are not.
Go requires that major versions after v1 involve a change in the import path of
the module. For example, version 2 of the module `gitlab.com/my/project` must be
imported and released as `gitlab.com/my/project/v2`.
For a complete understanding of Go modules and versioning, see [this series of
blog posts](https://blog.golang.org/using-go-modules) on the official Go
website.
## Valid modules and versions
The GitLab Go proxy will ignore modules and module versions that have an invalid
`module` directive in their `go.mod`. Go requires that a package imported as
`gitlab.com/my/project` can be accessed with that same URL, and that the first
line of `go.mod` is `module gitlab.com/my/project`. If `go.mod` names a
different module, compilation will fail. Additionally, Go requires, for major
versions after 1, that the name of the module have an appropriate suffix, for
example `gitlab.com/my/project/v2`. If the `module` directive does not also have
this suffix, compilation will fail.
Go supports 'pseudo-versions' that encode the timestamp and SHA of a commit.
Tags that match the pseudo-version pattern are ignored, as otherwise they could
interfere with fetching specific commits using a pseudo-version. Pseudo-versions
follow one of three formats:
- `vX.0.0-yyyymmddhhmmss-abcdefabcdef`, when no earlier tagged commit exists for X.
- `vX.Y.Z-pre.0.yyyymmddhhmmss-abcdefabcdef`, when most recent prior tag is vX.Y.Z-pre.
- `vX.Y.(Z+1)-0.yyyymmddhhmmss-abcdefabcdef`, when most recent prior tag is vX.Y.Z.
...@@ -21,6 +21,7 @@ The Packages feature allows GitLab to act as a repository for the following: ...@@ -21,6 +21,7 @@ The Packages feature allows GitLab to act as a repository for the following:
| [NPM Registry](npm_registry/index.md) **(PREMIUM)** | The GitLab NPM Registry enables every project in GitLab to have its own space to store [NPM](https://www.npmjs.com/) packages. | 11.7+ | | [NPM Registry](npm_registry/index.md) **(PREMIUM)** | The GitLab NPM Registry enables every project in GitLab to have its own space to store [NPM](https://www.npmjs.com/) packages. | 11.7+ |
| [NuGet Repository](nuget_repository/index.md) **(PREMIUM)** | The GitLab NuGet Repository will enable every project in GitLab to have its own space to store [NuGet](https://www.nuget.org/) packages. | 12.8+ | | [NuGet Repository](nuget_repository/index.md) **(PREMIUM)** | The GitLab NuGet Repository will enable every project in GitLab to have its own space to store [NuGet](https://www.nuget.org/) packages. | 12.8+ |
| [PyPi Repository](pypi_repository/index.md) **(PREMIUM)** | The GitLab PyPi Repository will enable every project in GitLab to have its own space to store [PyPi](https://pypi.org/) packages. | 12.10+ | | [PyPi Repository](pypi_repository/index.md) **(PREMIUM)** | The GitLab PyPi Repository will enable every project in GitLab to have its own space to store [PyPi](https://pypi.org/) packages. | 12.10+ |
| [Go Proxy](go_proxy/index.md) **(PREMIUM)** | The Go proxy for GitLab enables every project in GitLab to be fetched with the [Go proxy protocol](https://proxy.golang.org/). | 13.1+ |
## Enable the Package Registry for your project ## Enable the Package Registry for your project
...@@ -116,7 +117,6 @@ are adding support for [PHP](https://gitlab.com/gitlab-org/gitlab/-/merge_reques ...@@ -116,7 +117,6 @@ are adding support for [PHP](https://gitlab.com/gitlab-org/gitlab/-/merge_reques
| [Conda](https://gitlab.com/gitlab-org/gitlab/-/issues/36891) | Secure and private local Conda repositories. | | [Conda](https://gitlab.com/gitlab-org/gitlab/-/issues/36891) | Secure and private local Conda repositories. |
| [CRAN](https://gitlab.com/gitlab-org/gitlab/-/issues/36892) | Deploy and resolve CRAN packages for the R language. | | [CRAN](https://gitlab.com/gitlab-org/gitlab/-/issues/36892) | Deploy and resolve CRAN packages for the R language. |
| [Debian](https://gitlab.com/gitlab-org/gitlab/-/issues/5835) | Host and provision Debian packages. | | [Debian](https://gitlab.com/gitlab-org/gitlab/-/issues/5835) | Host and provision Debian packages. |
| [Go](https://gitlab.com/gitlab-org/gitlab/-/issues/9773) | Resolve Go dependencies from and publish your Go packages to GitLab. |
| [Opkg](https://gitlab.com/gitlab-org/gitlab/-/issues/36894) | Optimize your work with OpenWrt using Opkg repositories. | | [Opkg](https://gitlab.com/gitlab-org/gitlab/-/issues/36894) | Optimize your work with OpenWrt using Opkg repositories. |
| [P2](https://gitlab.com/gitlab-org/gitlab/-/issues/36895) | Host all your Eclipse plugins in your own GitLab P2 repository. | | [P2](https://gitlab.com/gitlab-org/gitlab/-/issues/36895) | Host all your Eclipse plugins in your own GitLab P2 repository. |
| [Puppet](https://gitlab.com/gitlab-org/gitlab/-/issues/36897) | Configuration management meets repository management with Puppet repositories. | | [Puppet](https://gitlab.com/gitlab-org/gitlab/-/issues/36897) | Configuration management meets repository management with Puppet repositories. |
......
# frozen_string_literal: true
module Packages
module Go
class ModuleFinder
include ::API::Helpers::Packages::Go::ModuleHelpers
attr_reader :project, :module_name
def initialize(project, module_name)
module_name = Pathname.new(module_name).cleanpath.to_s
@project = project
@module_name = module_name
end
def execute
return if @module_name.blank? || !@module_name.start_with?(gitlab_go_url)
module_path = @module_name[gitlab_go_url.length..].split('/')
project_path = project.full_path.split('/')
module_project_path = module_path.shift(project_path.length)
return unless module_project_path == project_path
Packages::GoModule.new(@project, @module_name, module_path.join('/'))
end
private
def gitlab_go_url
@gitlab_go_url ||= Settings.build_gitlab_go_url + '/'
end
end
end
end
# frozen_string_literal: true
module Packages
module Go
class VersionFinder
include ::API::Helpers::Packages::Go::ModuleHelpers
attr_reader :mod
def initialize(mod)
@mod = mod
end
def execute
@mod.project.repository.tags
.filter { |tag| semver? tag }
.map { |tag| @mod.version_by(ref: tag) }
.filter { |ver| ver.valid? }
end
def find(target)
case target
when String
if pseudo_version? target
semver = parse_semver(target)
commit = pseudo_version_commit(@mod.project, semver)
Packages::GoModuleVersion.new(@mod, :pseudo, commit, name: target, semver: semver)
else
@mod.version_by(ref: target)
end
when Gitlab::Git::Ref
@mod.version_by(ref: target)
when ::Commit, Gitlab::Git::Commit
@mod.version_by(commit: target)
else
raise ArgumentError.new 'not a valid target'
end
end
end
end
end
# frozen_string_literal: true
class Packages::GoModule
include Gitlab::Utils::StrongMemoize
attr_reader :project, :name, :path
def initialize(project, name, path)
@project = project
@name = name
@path = path
end
def versions
strong_memoize(:versions) { Packages::Go::VersionFinder.new(self).execute }
end
def version_by(ref: nil, commit: nil)
raise ArgumentError.new 'no filter specified' unless ref || commit
raise ArgumentError.new 'ref and commit are mutually exclusive' if ref && commit
if commit
return version_by_sha(commit) if commit.is_a? String
return version_by_commit(commit)
end
return version_by_name(ref) if ref.is_a? String
version_by_ref(ref)
end
def path_valid?(major)
m = /\/v(\d+)$/i.match(@name)
case major
when 0, 1
m.nil?
else
!m.nil? && m[1].to_i == major
end
end
def gomod_valid?(gomod)
if Feature.enabled?(:go_proxy_disable_gomod_validation, @project)
return gomod&.start_with?("module ")
end
gomod&.split("\n", 2)&.first == "module #{@name}"
end
private
def version_by_name(name)
# avoid a Gitaly call if possible
if strong_memoized?(:versions)
v = versions.find { |v| v.name == ref }
return v if v
end
ref = @project.repository.find_tag(name) || @project.repository.find_branch(name)
return unless ref
version_by_ref(ref)
end
def version_by_ref(ref)
# reuse existing versions
if strong_memoized?(:versions)
v = versions.find { |v| v.ref == ref }
return v if v
end
commit = ref.dereferenced_target
semver = Packages::SemVer.parse(ref.name, prefixed: true)
Packages::GoModuleVersion.new(self, :ref, commit, ref: ref, semver: semver)
end
def version_by_sha(sha)
commit = @project.commit_by(oid: sha)
return unless ref
version_by_commit(commit)
end
def version_by_commit(commit)
Packages::GoModuleVersion.new(self, :commit, commit)
end
end
# frozen_string_literal: true
class Packages::GoModuleVersion
include Gitlab::Utils::StrongMemoize
include ::API::Helpers::Packages::Go::ModuleHelpers
VALID_TYPES = %i[ref commit pseudo].freeze
attr_reader :mod, :type, :ref, :commit
delegate :major, to: :@semver, allow_nil: true
delegate :minor, to: :@semver, allow_nil: true
delegate :patch, to: :@semver, allow_nil: true
delegate :prerelease, to: :@semver, allow_nil: true
delegate :build, to: :@semver, allow_nil: true
def initialize(mod, type, commit, name: nil, semver: nil, ref: nil)
raise ArgumentError.new("invalid type '#{type}'") unless VALID_TYPES.include? type
raise ArgumentError.new("mod is required") unless mod
raise ArgumentError.new("commit is required") unless commit
if type == :ref
raise ArgumentError.new("ref is required") unless ref
elsif type == :pseudo
raise ArgumentError.new("name is required") unless name
raise ArgumentError.new("semver is required") unless semver
end
@mod = mod
@type = type
@commit = commit
@name = name if name
@semver = semver if semver
@ref = ref if ref
end
def name
@name || @ref&.name
end
def full_name
"#{mod.name}@#{name || commit.sha}"
end
def gomod
strong_memoize(:gomod) do
if strong_memoized?(:blobs)
blob_at(@mod.path + '/go.mod')
elsif @mod.path.empty?
@mod.project.repository.blob_at(@commit.sha, 'go.mod')&.data
else
@mod.project.repository.blob_at(@commit.sha, @mod.path + '/go.mod')&.data
end
end
end
def archive
suffix_len = @mod.path == '' ? 0 : @mod.path.length + 1
Zip::OutputStream.write_buffer do |zip|
files.each do |file|
zip.put_next_entry "#{full_name}/#{file[suffix_len...]}"
zip.write blob_at(file)
end
end
end
def files
strong_memoize(:files) do
ls_tree.filter { |e| !excluded.any? { |n| e.start_with? n } }
end
end
def excluded
strong_memoize(:excluded) do
ls_tree
.filter { |f| f.end_with?('/go.mod') && f != @mod.path + '/go.mod' }
.map { |f| f[0..-7] }
end
end
def valid?
@mod.path_valid?(major) && @mod.gomod_valid?(gomod)
end
private
def blob_at(path)
return if path.nil? || path.empty?
path = path[1..] if path.start_with? '/'
blobs.find { |x| x.path == path }&.data
end
def blobs
strong_memoize(:blobs) { @mod.project.repository.batch_blobs(files.map { |x| [@commit.sha, x] }) }
end
def ls_tree
strong_memoize(:ls_tree) do
path =
if @mod.path.empty?
'.'
else
@mod.path
end
@mod.project.repository.gitaly_repository_client.search_files_by_name(@commit.sha, path)
end
end
end
# frozen_string_literal: true
class Packages::SemVer
attr_accessor :major, :minor, :patch, :prerelease, :build
def initialize(major = 0, minor = 0, patch = 0, prerelease = nil, build = nil, prefixed: false)
@major = major
@minor = minor
@patch = patch
@prerelease = prerelease
@build = build
@prefixed = prefixed
end
def prefixed?
@prefixed
end
def ==(other)
self.class == other.class &&
self.major == other.major &&
self.minor == other.minor &&
self.patch == other.patch &&
self.prerelease == other.prerelease &&
self.build == other.build
end
def to_s
s = "#{prefixed? ? 'v' : ''}#{major || 0}.#{minor || 0}.#{patch || 0}"
s += "-#{prerelease}" if prerelease
s += "+#{build}" if build
s
end
def self.match(str, prefixed: false)
return unless str&.start_with?('v') == prefixed
str = str[1..] if prefixed
Gitlab::Regex.semver_regex.match(str)
end
def self.match?(str, prefixed: false)
!match(str, prefixed: prefixed).nil?
end
def self.parse(str, prefixed: false)
m = match str, prefixed: prefixed
return unless m
new(m[1].to_i, m[2].to_i, m[3].to_i, m[4], m[5], prefixed: prefixed)
end
end
# frozen_string_literal: true
module Packages
module Go
class ModuleVersionPresenter
def initialize(version)
@version = version
end
def name
@version.name
end
def time
@version.commit.committed_date
end
end
end
end
---
title: Implement Go module proxy MVC (package manager for Go)
merge_request: 27746
author: Ethan Reesor
type: added
# frozen_string_literal: true
module API
class GoProxy < Grape::API
helpers ::API::Helpers::PackagesHelpers
helpers ::API::Helpers::Packages::Go::ModuleHelpers
# basic semver, except case encoded (A => !a)
MODULE_VERSION_REGEX = /v(0|[1-9]\d*)\.(0|[1-9]\d*)\.(0|[1-9]\d*)(?:-([-.!a-z0-9]+))?(?:\+([-.!a-z0-9]+))?/.freeze
MODULE_VERSION_REQUIREMENTS = { module_version: MODULE_VERSION_REGEX }.freeze
before { require_packages_enabled! }
helpers do
def find_project!(id)
# based on API::Helpers::Packages::BasicAuthHelpers#authorized_project_find!
project = find_project(id)
return project if project && can?(current_user, :read_project, project)
if current_user
not_found!('Project')
else
unauthorized!
end
end
def find_module
not_found! unless Feature.enabled?(:go_proxy, user_project)
module_name = case_decode params[:module_name]
bad_request!('Module Name') if module_name.blank?
mod = ::Packages::Go::ModuleFinder.new(user_project, module_name).execute
not_found! unless mod
mod
end
def find_version
module_version = case_decode params[:module_version]
ver = ::Packages::Go::VersionFinder.new(find_module).find(module_version)
not_found! unless ver&.valid?
ver
rescue ArgumentError
not_found!
end
end
params do
requires :id, type: String, desc: 'The ID of a project'
requires :module_name, type: String, desc: 'Module name', coerce_with: ->(val) { CGI.unescape(val) }
end
route_setting :authentication, job_token_allowed: true, basic_auth_personal_access_token: true
resource :projects, requirements: API::NAMESPACE_OR_PROJECT_REQUIREMENTS do
before do
authorize_read_package!
authorize_packages_feature!
end
namespace ':id/packages/go/*module_name/@v' do
desc 'Get all tagged versions for a given Go module' do
detail 'See `go help goproxy`, GET $GOPROXY/<module>/@v/list. This feature was introduced in GitLab 13.1.'
end
get 'list' do
mod = find_module
content_type 'text/plain'
mod.versions.map { |t| t.name }.join("\n")
end
desc 'Get information about the given module version' do
detail 'See `go help goproxy`, GET $GOPROXY/<module>/@v/<version>.info. This feature was introduced in GitLab 13.1.'
success EE::API::Entities::GoModuleVersion
end
params do
requires :module_version, type: String, desc: 'Module version'
end
get ':module_version.info', requirements: MODULE_VERSION_REQUIREMENTS do
ver = find_version
present ::Packages::Go::ModuleVersionPresenter.new(ver), with: EE::API::Entities::GoModuleVersion
end
desc 'Get the module file of the given module version' do
detail 'See `go help goproxy`, GET $GOPROXY/<module>/@v/<version>.mod. This feature was introduced in GitLab 13.1.'
end
params do
requires :module_version, type: String, desc: 'Module version'
end
get ':module_version.mod', requirements: MODULE_VERSION_REQUIREMENTS do
ver = find_version
content_type 'text/plain'
ver.gomod
end
desc 'Get a zip of the source of the given module version' do
detail 'See `go help goproxy`, GET $GOPROXY/<module>/@v/<version>.zip. This feature was introduced in GitLab 13.1.'
end
params do
requires :module_version, type: String, desc: 'Module version'
end
get ':module_version.zip', requirements: MODULE_VERSION_REQUIREMENTS do
ver = find_version
content_type 'application/zip'
env['api.format'] = :binary
header['Content-Disposition'] = ActionDispatch::Http::ContentDisposition.format(disposition: 'attachment', filename: ver.name + '.zip')
header['Content-Transfer-Encoding'] = 'binary'
status :ok
body ver.archive.string
end
end
end
end
end
# frozen_string_literal: true
module API
module Helpers
module Packages
module Go
module ModuleHelpers
def case_encode(str)
# Converts "github.com/Azure" to "github.com/!azure"
#
# From `go help goproxy`:
#
# > To avoid problems when serving from case-sensitive file systems,
# > the <module> and <version> elements are case-encoded, replacing
# > every uppercase letter with an exclamation mark followed by the
# > corresponding lower-case letter: github.com/Azure encodes as
# > github.com/!azure.
str.gsub(/A-Z/) { |s| "!#{s.downcase}"}
end
def case_decode(str)
# Converts "github.com/!azure" to "github.com/Azure"
#
# See #case_encode
str.gsub(/![[:alpha:]]/) { |s| s[1..].upcase }
end
def semver?(tag)
return false if tag.dereferenced_target.nil?
::Packages::SemVer.match?(tag.name, prefixed: true)
end
def pseudo_version?(version)
return false unless version
if version.is_a? String
version = parse_semver version
return false unless version
end
pre = version.prerelease
# Valid pseudo-versions are:
# vX.0.0-yyyymmddhhmmss-sha1337beef0, when no earlier tagged commit exists for X
# vX.Y.Z-pre.0.yyyymmddhhmmss-sha1337beef0, when most recent prior tag is vX.Y.Z-pre
# vX.Y.(Z+1)-0.yyyymmddhhmmss-sha1337beef0, when most recent prior tag is vX.Y.Z
if version.minor != 0 || version.patch != 0
m = /\A(.*\.)?0\./.freeze.match pre
return false unless m
pre = pre[m[0].length..]
end
# This pattern is intentionally more forgiving than the patterns
# above. Correctness is verified by #pseudo_version_commit.
/\A\d{14}-\h+\z/.freeze.match? pre
end
def pseudo_version_commit(project, semver)
# Per Go's implementation of pseudo-versions, a tag should be
# considered a pseudo-version if it matches one of the patterns
# listed in #pseudo_version?, regardless of the content of the
# timestamp or the length of the SHA fragment. However, an error
# should be returned if the timestamp is not correct or if the SHA
# fragment is not exactly 12 characters long. See also Go's
# implementation of:
#
# - [*codeRepo.validatePseudoVersion](https://github.com/golang/go/blob/daf70d6c1688a1ba1699c933b3c3f04d6f2f73d9/src/cmd/go/internal/modfetch/coderepo.go#L530)
# - [Pseudo-version parsing](https://github.com/golang/go/blob/master/src/cmd/go/internal/modfetch/pseudo.go)
# - [Pseudo-version request processing](https://github.com/golang/go/blob/master/src/cmd/go/internal/modfetch/coderepo.go)
# Go ignores anything before '.' or after the second '-', so we will do the same
timestamp, sha = semver.prerelease.split('-').last 2
timestamp = timestamp.split('.').last
commit = project.repository.commit_by(oid: sha)
# Error messages are based on the responses of proxy.golang.org
# Verify that the SHA fragment references a commit
raise ArgumentError.new 'invalid pseudo-version: unknown commit' unless commit
# Require the SHA fragment to be 12 characters long
raise ArgumentError.new 'invalid pseudo-version: revision is shorter than canonical' unless sha.length == 12
# Require the timestamp to match that of the commit
raise ArgumentError.new 'invalid pseudo-version: does not match version-control timestamp' unless commit.committed_date.strftime('%Y%m%d%H%M%S') == timestamp
commit
end
def parse_semver(str)
::Packages::SemVer.parse(str, prefixed: true)
end
end
end
end
end
end
...@@ -38,6 +38,7 @@ module EE ...@@ -38,6 +38,7 @@ module EE
mount ::API::ConanPackages mount ::API::ConanPackages
mount ::API::MavenPackages mount ::API::MavenPackages
mount ::API::NpmPackages mount ::API::NpmPackages
mount ::API::GoProxy
mount ::API::MergeTrains mount ::API::MergeTrains
mount ::API::ProjectPackages mount ::API::ProjectPackages
mount ::API::GroupPackages mount ::API::GroupPackages
......
# frozen_string_literal: true
module EE
module API
module Entities
class GoModuleVersion < Grape::Entity
expose :name, as: 'Version'
expose :time, as: 'Time'
end
end
end
end
# frozen_string_literal: true
FactoryBot.define do
factory :go_module_commit, class: 'Commit' do
skip_create
transient do
project { raise ArgumentError.new("project is required") }
service { raise ArgumentError.new("this factory cannot be used without specifying a trait") }
tag { nil }
tag_message { nil }
commit do
r = service.execute
raise "operation failed: #{r}" unless r[:status] == :success
commit = project.repository.commit_by(oid: r[:result])
if tag
r = Tags::CreateService.new(project, project.owner).execute(tag, commit.sha, tag_message)
raise "operation failed: #{r}" unless r[:status] == :success
end
commit
end
end
initialize_with do
commit
end
trait :files do
transient do
files { raise ArgumentError.new("files is required") }
message { 'Add files' }
end
service do
Files::MultiService.new(
project,
project.owner,
commit_message: message,
start_branch: project.repository.root_ref || 'master',
branch_name: project.repository.root_ref || 'master',
actions: files.map do |path, content|
{ action: :create, file_path: path, content: content }
end
)
end
end
trait :package do
transient do
path { raise ArgumentError.new("path is required") }
message { 'Add package' }
end
service do
Files::MultiService.new(
project,
project.owner,
commit_message: message,
start_branch: project.repository.root_ref || 'master',
branch_name: project.repository.root_ref || 'master',
actions: [
{ action: :create, file_path: path + '/b.go', content: "package b\nfunc Bye() { println(\"Goodbye world!\") }\n" }
]
)
end
end
trait :module do
transient do
name { nil }
message { 'Add module' }
url do
v = "#{::Gitlab.config.gitlab.host}/#{project.path_with_namespace}"
if name
v + '/' + name
else
v
end
end
path do
if name
name + '/'
else
''
end
end
end
service do
Files::MultiService.new(
project,
project.owner,
commit_message: message,
start_branch: project.repository.root_ref || 'master',
branch_name: project.repository.root_ref || 'master',
actions: [
{ action: :create, file_path: path + 'go.mod', content: "module #{url}\n" },
{ action: :create, file_path: path + 'a.go', content: "package a\nfunc Hi() { println(\"Hello world!\") }\n" }
]
)
end
end
end
end
# frozen_string_literal: true
FactoryBot.define do
factory :go_module_version, class: 'Packages::GoModuleVersion' do
skip_create
initialize_with do
p = attributes[:params]
s = Packages::SemVer.parse(p.semver, prefixed: true)
raise ArgumentError.new("invalid sematic version: '#{p.semver}''") if !s && p.semver
new(p.mod, p.type, p.commit, name: p.name, semver: s, ref: p.ref)
end
mod { create :go_module }
type { :commit }
commit { raise ArgumentError.new("commit is required") }
name { nil }
semver { nil }
ref { nil }
params { OpenStruct.new(mod: mod, type: type, commit: commit, name: name, semver: semver, ref: ref) }
trait :tagged do
name { raise ArgumentError.new("name is required") }
ref { mod.project.repository.find_tag(name) }
commit { ref.dereferenced_target }
params { OpenStruct.new(mod: mod, type: :ref, commit: commit, semver: name, ref: ref) }
end
trait :pseudo do
transient do
prefix { raise ArgumentError.new("prefix is required") }
end
type { :pseudo }
name { "#{prefix}#{commit.committed_date.strftime('%Y%m%d%H%M%S')}-#{commit.sha[0..11]}" }
params { OpenStruct.new(mod: mod, type: :pseudo, commit: commit, name: name, semver: name) }
end
end
end
# frozen_string_literal: true
FactoryBot.define do
factory :go_module, class: 'Packages::GoModule' do
initialize_with { new(attributes[:project], attributes[:name], attributes[:path]) }
skip_create
project
path { '' }
name { "#{Settings.build_gitlab_go_url}/#{project.full_path}#{path.empty? ? '' : '/'}#{path}" }
end
end
# frozen_string_literal: true
require 'spec_helper'
describe Packages::Go::ModuleFinder do
let_it_be(:project) { create :project }
let_it_be(:other_project) { create :project }
let(:finder) { described_class.new project, module_name }
shared_examples 'an invalid path' do
describe '#module_name' do
it 'returns the expected name' do
expect(finder.module_name).to eq(expected_name)
end
end
describe '#execute' do
it 'returns nil' do
expect(finder.execute).to be_nil
end
end
end
describe '#execute' do
context 'with module name equal to project name' do
let(:module_name) { base_url(project) }
it 'returns a module with empty path' do
mod = finder.execute
expect(mod).not_to be_nil
expect(mod.path).to eq('')
end
end
context 'with module name starting with project name and slash' do
let(:module_name) { base_url(project) + '/mod' }
it 'returns a module with non-empty path' do
mod = finder.execute
expect(mod).not_to be_nil
expect(mod.path).to eq('mod')
end
end
context 'with a module name not equal to and not starting with project name' do
let(:module_name) { base_url(other_project) }
it 'returns nil' do
expect(finder.execute).to be_nil
end
end
end
context 'with relative path component' do
it_behaves_like 'an invalid path' do
let(:module_name) { base_url(project) + '/../xyz' }
let(:expected_name) { base_url(project.namespace) + '/xyz' }
end
end
context 'with many relative path components' do
it_behaves_like 'an invalid path' do
let(:module_name) { base_url(project) + ('/..' * 10) + '/xyz' }
let(:expected_name) { ('../' * 7) + 'xyz' }
end
end
def base_url(project)
"#{Settings.build_gitlab_go_url}/#{project.full_path}"
end
end
# frozen_string_literal: true
require 'spec_helper'
describe Packages::Go::VersionFinder do
let_it_be(:user) { create :user }
let_it_be(:project) { create :project_empty_repo, creator: user, path: 'my-go-lib' }
let(:finder) { described_class.new mod }
before :all do
create :go_module_commit, :files, project: project, tag: 'v1.0.0', files: { 'README.md' => 'Hi' }
create :go_module_commit, :module, project: project, tag: 'v1.0.1'
create :go_module_commit, :package, project: project, tag: 'v1.0.2', path: 'pkg'
create :go_module_commit, :module, project: project, tag: 'v1.0.3', name: 'mod'
create :go_module_commit, :module, project: project, tag: 'v1.0.4', name: 'bad-mod', url: 'example.com/go-lib'
create :go_module_commit, :files, project: project, tag: 'c1', files: { 'y.go' => "package a\n" }
create :go_module_commit, :module, project: project, tag: 'c2', name: 'v2'
create :go_module_commit, :files, project: project, tag: 'v2.0.0', files: { 'v2/x.go' => "package a\n" }
end
before do
stub_feature_flags(go_proxy_disable_gomod_validation: false)
end
shared_examples '#execute' do |*expected|
it "returns #{expected.empty? ? 'nothing' : expected.join(', ')}" do
actual = finder.execute.map { |x| x.name }
expect(actual.to_set).to eq(expected.to_set)
end
end
shared_examples '#find with an invalid argument' do |message|
it "raises an argument exception: #{message}" do
expect { finder.find(target) }.to raise_error(ArgumentError, message)
end
end
describe '#execute' do
context 'for the root module' do
let(:mod) { create :go_module, project: project }
it_behaves_like '#execute', 'v1.0.1', 'v1.0.2', 'v1.0.3', 'v1.0.4'
end
context 'for the package' do
let(:mod) { create :go_module, project: project, path: 'pkg' }
it_behaves_like '#execute'
end
context 'for the submodule' do
let(:mod) { create :go_module, project: project, path: 'mod' }
it_behaves_like '#execute', 'v1.0.3', 'v1.0.4'
end
context 'for the root module v2' do
let(:mod) { create :go_module, project: project, path: 'v2' }
it_behaves_like '#execute', 'v2.0.0'
end
context 'for the bad module' do
let(:mod) { create :go_module, project: project, path: 'bad-mod' }
context 'with gomod checking enabled' do
it_behaves_like '#execute'
end
context 'with gomod checking disabled' do
before do
stub_feature_flags(go_proxy_disable_gomod_validation: true)
end
it_behaves_like '#execute', 'v1.0.4'
end
end
end
describe '#find' do
let(:mod) { create :go_module, project: project }
context 'with a ref' do
it 'returns a ref version' do
ref = project.repository.find_branch 'master'
v = finder.find(ref)
expect(v.type).to eq(:ref)
expect(v.ref).to eq(ref)
end
end
context 'with a semver tag' do
it 'returns a version with a semver' do
v = finder.find(project.repository.find_tag('v1.0.0'))
expect(v.major).to eq(1)
expect(v.minor).to eq(0)
expect(v.patch).to eq(0)
expect(v.prerelease).to be_nil
expect(v.build).to be_nil
end
end
context 'with a semver tag string' do
it 'returns a version with a semver' do
v = finder.find('v1.0.1')
expect(v.major).to eq(1)
expect(v.minor).to eq(0)
expect(v.patch).to eq(1)
expect(v.prerelease).to be_nil
expect(v.build).to be_nil
end
end
context 'with a commit' do
it 'retruns a commit version' do
v = finder.find(project.repository.head_commit)
expect(v.type).to eq(:commit)
end
end
context 'with a pseudo-version' do
it 'returns a pseudo version' do
commit = project.repository.head_commit
pseudo = "v0.0.0-#{commit.committed_date.strftime('%Y%m%d%H%M%S')}-#{commit.sha[0..11]}"
v = finder.find(pseudo)
expect(v.type).to eq(:pseudo)
expect(v.commit).to eq(commit)
expect(v.name).to eq(pseudo)
end
end
context 'with a string that is not a semantic version' do
it 'returns nil' do
expect(finder.find('not-a-semver')).to be_nil
end
end
context 'with a pseudo-version that does not reference a commit' do
it_behaves_like '#find with an invalid argument', 'invalid pseudo-version: unknown commit' do
let(:commit) { project.repository.head_commit }
let(:target) { "v0.0.0-#{commit.committed_date.strftime('%Y%m%d%H%M%S')}-#{'0' * 12}" }
end
end
context 'with a pseudo-version with a short sha' do
it_behaves_like '#find with an invalid argument', 'invalid pseudo-version: revision is shorter than canonical' do
let(:commit) { project.repository.head_commit }
let(:target) { "v0.0.0-#{commit.committed_date.strftime('%Y%m%d%H%M%S')}-#{commit.sha[0..10]}" }
end
end
context 'with a pseudo-version with an invalid timestamp' do
it_behaves_like '#find with an invalid argument', 'invalid pseudo-version: does not match version-control timestamp' do
let(:commit) { project.repository.head_commit }
let(:target) { "v0.0.0-#{'0' * 14}-#{commit.sha[0..11]}" }
end
end
end
end
# frozen_string_literal: true
require 'spec_helper'
describe Packages::GoModule, type: :model do
before do
stub_feature_flags(go_proxy_disable_gomod_validation: false)
end
describe '#path_valid?' do
context 'with root path' do
let_it_be(:package) { create(:go_module) }
context 'with major version 0' do
it('returns true') { expect(package.path_valid?(0)).to eq(true) }
end
context 'with major version 1' do
it('returns true') { expect(package.path_valid?(1)).to eq(true) }
end
context 'with major version 2' do
it('returns false') { expect(package.path_valid?(2)).to eq(false) }
end
end
context 'with path ./v2' do
let_it_be(:package) { create(:go_module, path: '/v2') }
context 'with major version 0' do
it('returns false') { expect(package.path_valid?(0)).to eq(false) }
end
context 'with major version 1' do
it('returns false') { expect(package.path_valid?(1)).to eq(false) }
end
context 'with major version 2' do
it('returns true') { expect(package.path_valid?(2)).to eq(true) }
end
end
end
describe '#gomod_valid?' do
let_it_be(:package) { create(:go_module) }
context 'with good gomod' do
it('returns true') { expect(package.gomod_valid?("module #{package.name}")).to eq(true) }
end
context 'with bad gomod' do
it('returns false') { expect(package.gomod_valid?("module #{package.name}/v2")).to eq(false) }
end
context 'with empty gomod' do
it('returns false') { expect(package.gomod_valid?("")).to eq(false) }
end
end
end
# frozen_string_literal: true
require 'spec_helper'
describe Packages::GoModuleVersion, type: :model do
let_it_be(:user) { create :user }
let_it_be(:project) { create :project_empty_repo, creator: user, path: 'my-go-lib' }
let_it_be(:mod) { create :go_module, project: project }
before :all do
create :go_module_commit, :files, project: project, tag: 'v1.0.0', files: { 'README.md' => 'Hi' }
create :go_module_commit, :module, project: project, tag: 'v1.0.1'
create :go_module_commit, :package, project: project, tag: 'v1.0.2', path: 'pkg'
create :go_module_commit, :module, project: project, tag: 'v1.0.3', name: 'mod'
create :go_module_commit, :files, project: project, files: { 'y.go' => "package a\n" }
create :go_module_commit, :module, project: project, name: 'v2'
create :go_module_commit, :files, project: project, tag: 'v2.0.0', files: { 'v2/x.go' => "package a\n" }
end
shared_examples '#files' do |desc, *entries|
it "returns #{desc}" do
actual = version.files.map { |x| x }.to_set
expect(actual).to eq(entries.to_set)
end
end
shared_examples '#archive' do |desc, *entries|
it "returns an archive of #{desc}" do
expected = entries.map { |e| "#{version.full_name}/#{e}" }.to_set
actual = Set[]
Zip::InputStream.open(StringIO.new(version.archive.string)) do |zip|
while (entry = zip.get_next_entry)
actual.add(entry.name)
end
end
expect(actual).to eq(expected)
end
end
describe '#name' do
context 'with ref and name specified' do
let_it_be(:version) { create :go_module_version, mod: mod, name: 'foobar', commit: project.repository.head_commit, ref: project.repository.find_tag('v1.0.0') }
it('returns that name') { expect(version.name).to eq('foobar') }
end
context 'with ref specified and name unspecified' do
let_it_be(:version) { create :go_module_version, mod: mod, commit: project.repository.head_commit, ref: project.repository.find_tag('v1.0.0') }
it('returns the name of the ref') { expect(version.name).to eq('v1.0.0') }
end
context 'with ref and name unspecified' do
let_it_be(:version) { create :go_module_version, mod: mod, commit: project.repository.head_commit }
it('returns nil') { expect(version.name).to eq(nil) }
end
end
describe '#gomod' do
context 'with go.mod missing' do
let_it_be(:version) { create :go_module_version, :tagged, mod: mod, name: 'v1.0.0' }
it('returns nil') { expect(version.gomod).to eq(nil) }
end
context 'with go.mod present' do
let_it_be(:version) { create :go_module_version, :tagged, mod: mod, name: 'v1.0.1' }
it('returns the contents of go.mod') { expect(version.gomod).to eq("module #{mod.name}\n") }
end
end
describe '#files' do
context 'with a root module' do
context 'with an empty module path' do
let_it_be(:version) { create :go_module_version, :tagged, mod: mod, name: 'v1.0.2' }
it_behaves_like '#files', 'all the files', 'README.md', 'go.mod', 'a.go', 'pkg/b.go'
end
end
context 'with a root module and a submodule' do
context 'with an empty module path' do
let_it_be(:version) { create :go_module_version, :tagged, mod: mod, name: 'v1.0.3' }
it_behaves_like '#files', 'files excluding the submodule', 'README.md', 'go.mod', 'a.go', 'pkg/b.go'
end
context 'with the submodule\'s path' do
let_it_be(:mod) { create :go_module, project: project, path: 'mod' }
let_it_be(:version) { create :go_module_version, :tagged, mod: mod, name: 'v1.0.3' }
it_behaves_like '#files', 'the submodule\'s files', 'mod/go.mod', 'mod/a.go'
end
end
end
describe '#archive' do
context 'with a root module' do
context 'with an empty module path' do
let_it_be(:version) { create :go_module_version, :tagged, mod: mod, name: 'v1.0.2' }
it_behaves_like '#archive', 'all the files', 'README.md', 'go.mod', 'a.go', 'pkg/b.go'
end
end
context 'with a root module and a submodule' do
context 'with an empty module path' do
let_it_be(:version) { create :go_module_version, :tagged, mod: mod, name: 'v1.0.3' }
it_behaves_like '#archive', 'files excluding the submodule', 'README.md', 'go.mod', 'a.go', 'pkg/b.go'
end
context 'with the submodule\'s path' do
let_it_be(:mod) { create :go_module, project: project, path: 'mod' }
let_it_be(:version) { create :go_module_version, :tagged, mod: mod, name: 'v1.0.3' }
it_behaves_like '#archive', 'the submodule\'s files', 'go.mod', 'a.go'
end
end
end
end
# frozen_string_literal: true
require 'spec_helper'
describe Packages::SemVer, type: :model do
shared_examples '#parse with a valid semver' do |str, major, minor, patch, prerelease, build|
context "with #{str}" do
it "returns #{described_class.new(major, minor, patch, prerelease, build, prefixed: true)} with prefix" do
expected = described_class.new(major, minor, patch, prerelease, build, prefixed: true)
expect(described_class.parse('v' + str, prefixed: true)).to eq(expected)
end
it "returns #{described_class.new(major, minor, patch, prerelease, build)} without prefix" do
expected = described_class.new(major, minor, patch, prerelease, build)
expect(described_class.parse(str)).to eq(expected)
end
end
end
shared_examples '#parse with an invalid semver' do |str|
context "with #{str}" do
it 'returns nil with prefix' do
expect(described_class.parse('v' + str, prefixed: true)).to be_nil
end
it 'returns nil without prefix' do
expect(described_class.parse(str)).to be_nil
end
end
end
describe '#parse' do
it_behaves_like '#parse with a valid semver', '1.0.0', 1, 0, 0, nil, nil
it_behaves_like '#parse with a valid semver', '1.0.0-pre', 1, 0, 0, 'pre', nil
it_behaves_like '#parse with a valid semver', '1.0.0+build', 1, 0, 0, nil, 'build'
it_behaves_like '#parse with a valid semver', '1.0.0-pre+build', 1, 0, 0, 'pre', 'build'
it_behaves_like '#parse with an invalid semver', '01.0.0'
it_behaves_like '#parse with an invalid semver', '0.01.0'
it_behaves_like '#parse with an invalid semver', '0.0.01'
it_behaves_like '#parse with an invalid semver', '1.0.0asdf'
end
end
This diff is collapsed.
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment