Skip to content

G
gitlab-ce
Commit 3165509c authored by Igor Drozdov's avatar Igor Drozdov
Browse files
Options

Merge branch '354845-aqualls-move' into 'master' 

Move workhorse files into final directory

See merge request gitlab-org/gitlab!84115
parents 5dcdce4e ad9aa486
Show whitespace changes
Inline Side-by-side
Showing with 680 additions and 592 deletions
doc/development/backend/create_source_code_be/index.md
View file @ 3165509c
......@@ -21,14 +21,12 @@ The team works across three codebases: Workhorse, GitLab Shell and GitLab Rails.
## Workhorse
GitLab Workhorse is a smart reverse proxy for GitLab. It handles "large" HTTP
[GitLab Workhorse](../../workhorse/index.md) is a smart reverse proxy for GitLab. It handles "large" HTTP
requests such as file downloads, file uploads, `git push`, `git pull` and `git` archive downloads.
Workhorse itself is not a feature, but there are several features in GitLab
that would not work efficiently without Workhorse.
Workhorse documentation is available in the [Workhorse repository](https://gitlab.com/gitlab-org/gitlab/tree/master/workhorse).
## GitLab Shell
GitLab Shell handles Git SSH sessions for GitLab and modifies the list of authorized keys.
......
doc/development/workhorse/channel.md 0 → 100644
View file @ 3165509c
---
stage: Create
group: Source Code
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments
---
# Websocket channel support
In some cases, GitLab can provide in-browser terminal access to an
environment (which is a running server or container, onto which a
project has been deployed), or even access to services running in CI
through a WebSocket. Workhorse manages the WebSocket upgrade and
long-lived connection to the websocket connection, which frees
up GitLab to process other requests.
This document outlines the architecture of these connections.
## Introduction to WebSockets
A websocket is an "upgraded" HTTP/1.1 request. Their purpose is to
permit bidirectional communication between a client and a server.
**Websockets are not HTTP**. Clients can send messages (known as
frames) to the server at any time, and vice-versa. Client messages
are not necessarily requests, and server messages are not necessarily
responses. WebSocket URLs have schemes like `ws://` (unencrypted) or
`wss://` (TLS-secured).
When requesting an upgrade to WebSocket, the browser sends a HTTP/1.1
request that looks like this:
```plaintext
GET /path.ws HTTP/1.1
Connection: upgrade
Upgrade: websocket
Sec-WebSocket-Protocol: terminal.gitlab.com
# More headers, including security measures
```
At this point, the connection is still HTTP, so this is a request and
the server can send a normal HTTP response, including `404 Not Found`,
`500 Internal Server Error`, etc.
If the server decides to permit the upgrade, it will send a HTTP
`101 Switching Protocols` response. From this point, the connection
is no longer HTTP. It is a WebSocket and frames, not HTTP requests,
will flow over it. The connection will persist until the client or
server closes the connection.
In addition to the subprotocol, individual websocket frames may
also specify a message type - examples include `BinaryMessage`,
`TextMessage`, `Ping`, `Pong` or `Close`. Only binary frames can
contain arbitrary data - other frames are expected to be valid
UTF-8 strings, in addition to any subprotocol expectations.
## Browser to Workhorse
Using the terminal as an example, GitLab serves a JavaScript terminal
emulator to the browser on a URL like
`https://gitlab.com/group/project/-/environments/1/terminal`.
This opens a websocket connection to, e.g.,
`wss://gitlab.com/group/project/-/environments/1/terminal.ws`,
This endpoint doesn't exist in GitLab - only in Workhorse.
When receiving the connection, Workhorse first checks that the
client is authorized to access the requested terminal. It does
this by performing a "preauthentication" request to GitLab.
If the client has the appropriate permissions and the terminal
exists, GitLab responds with a successful response that includes
details of the terminal that the client should be connected to.
Otherwise, it returns an appropriate HTTP error response.
Errors are passed back to the client as HTTP responses, but if
GitLab returns valid terminal details to Workhorse, it will
connect to the specified terminal, upgrade the browser to a
WebSocket, and proxy between the two connections for as long
as the browser's credentials are valid. Workhorse will also
send regular `PingMessage` control frames to the browser, to
keep intervening proxies from terminating the connection
while the browser is present.
The browser must request an upgrade with a specific subprotocol:
### `terminal.gitlab.com`
This subprotocol considers `TextMessage` frames to be invalid.
Control frames, such as `PingMessage` or `CloseMessage`, have
their usual meanings.
`BinaryMessage` frames sent from the browser to the server are
arbitrary text input.
`BinaryMessage` frames sent from the server to the browser are
arbitrary text output.
These frames are expected to contain ANSI text control codes
and may be in any encoding.
### `base64.terminal.gitlab.com`
This subprotocol considers `BinaryMessage` frames to be invalid.
Control frames, such as `PingMessage` or `CloseMessage`, have
their usual meanings.
`TextMessage` frames sent from the browser to the server are
base64-encoded arbitrary text input (so the server must
base64-decode them before inputting them).
`TextMessage` frames sent from the server to the browser are
base64-encoded arbitrary text output (so the browser must
base64-decode them before outputting them).
In their base64-encoded form, these frames are expected to
contain ANSI terminal control codes, and may be in any encoding.
## Workhorse to GitLab
Using again the terminal as an example, before upgrading the browser,
Workhorse sends a normal HTTP request to GitLab on a URL like
`https://gitlab.com/group/project/environments/1/terminal.ws/authorize`.
This returns a JSON response containing details of where the
terminal can be found, and how to connect it. In particular,
the following details are returned in case of success:
- WebSocket URL to **connect** to, e.g.: `wss://example.com/terminals/1.ws?tty=1`
- WebSocket subprotocols to support, e.g.: `["channel.k8s.io"]`
- Headers to send, e.g.: `Authorization: Token xxyyz..`
- Certificate authority to verify `wss` connections with (optional)
Workhorse periodically re-checks this endpoint, and if it gets an
error response, or the details of the terminal change, it will
terminate the websocket session.
## Workhorse to the WebSocket server
In GitLab, environments or CI jobs may have a deployment service (e.g.,
`KubernetesService`) associated with them. This service knows
where the terminals or the service for an environment may be found, and these
details are returned to Workhorse by GitLab.
These URLs are *also* WebSocket URLs, and GitLab tells Workhorse
which subprotocols to speak over the connection, along with any
authentication details required by the remote end.
Before upgrading the browser's connection to a websocket,
Workhorse opens a HTTP client connection, according to the
details given to it by Workhorse, and attempts to upgrade
that connection to a websocket. If it fails, an error
response is sent to the browser; otherwise, the browser is
also upgraded.
Workhorse now has two websocket connections, albeit with
differing subprotocols. It decodes incoming frames from the
browser, re-encodes them to the channel's subprotocol, and
sends them to the channel. Similarly, it decodes incoming
frames from the channel, re-encodes them to the browser's
subprotocol, and sends them to the browser.
When either connection closes or enters an error state,
Workhorse detects the error and closes the other connection,
terminating the channel session. If the browser is the
connection that has disconnected, Workhorse will send an ANSI
`End of Transmission` control code (the `0x04` byte) to the
channel, encoded according to the appropriate subprotocol.
Workhorse will automatically reply to any websocket ping frame
sent by the channel, to avoid being disconnected.
Currently, Workhorse only supports the following subprotocols.
Supporting new deployment services will require new subprotocols
to be supported:
### `channel.k8s.io`
Used by Kubernetes, this subprotocol defines a simple multiplexed
channel.
Control frames have their usual meanings. `TextMessage` frames are
invalid. `BinaryMessage` frames represent I/O to a specific file
descriptor.
The first byte of each `BinaryMessage` frame represents the file
descriptor (fd) number, as a `uint8` (so the value `0x00` corresponds
to fd 0, `STDIN`, while `0x01` corresponds to fd 1, `STDOUT`).
The remaining bytes represent arbitrary data. For frames received
from the server, they are bytes that have been received from that
fd. For frames sent to the server, they are bytes that should be
written to that fd.
### `base64.channel.k8s.io`
Also used by Kubernetes, this subprotocol defines a similar multiplexed
channel to `channel.k8s.io`. The main differences are:
- `TextMessage` frames are valid, rather than `BinaryMessage` frames.
- The first byte of each `TextMessage` frame represents the file
descriptor as a numeric UTF-8 character, so the character `U+0030`,
or "0", is fd 0, STDIN).
- The remaining bytes represent base64-encoded arbitrary data.
doc/development/workhorse/configuration.md 0 → 100644
View file @ 3165509c
---
stage: Create
group: Source Code
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments
---
# Workhorse configuration
For historical reasons Workhorse uses both command line flags, a configuration file and environment variables.
All new configuration options that get added to Workhorse should go into the configuration file.
## CLI options
```plaintext
gitlab-workhorse [OPTIONS]
Options:
-apiCiLongPollingDuration duration
Long polling duration for job requesting for runners (default 50ns)
-apiLimit uint
Number of API requests allowed at single time
-apiQueueDuration duration
Maximum queueing duration of requests (default 30s)
-apiQueueLimit uint
Number of API requests allowed to be queued
-authBackend string
Authentication/authorization backend (default "http://localhost:8080")
-authSocket string
Optional: Unix domain socket to dial authBackend at
-cableBackend string
Optional: ActionCable backend (default authBackend)
-cableSocket string
Optional: Unix domain socket to dial cableBackend at (default authSocket)
-config string
TOML file to load config from
-developmentMode
Allow the assets to be served from Rails app
-documentRoot string
Path to static files content (default "public")
-listenAddr string
Listen address for HTTP server (default "localhost:8181")
-listenNetwork string
Listen 'network' (tcp, tcp4, tcp6, unix) (default "tcp")
-listenUmask int
Umask for Unix socket
-logFile string
Log file location
-logFormat string
Log format to use defaults to text (text, json, structured, none) (default "text")
-pprofListenAddr string
pprof listening address, e.g. 'localhost:6060'
-prometheusListenAddr string
Prometheus listening address, e.g. 'localhost:9229'
-proxyHeadersTimeout duration
How long to wait for response headers when proxying the request (default 5m0s)
-secretPath string
File with secret key to authenticate with authBackend (default "./.gitlab_workhorse_secret")
-version
Print version and exit
```
The 'auth backend' refers to the GitLab Rails application. The name is
a holdover from when GitLab Workhorse only handled Git push/pull over
HTTP.
GitLab Workhorse can listen on either a TCP or a Unix domain socket. It
can also open a second listening TCP listening socket with the Go
[`net/http/pprof` profiler server](http://golang.org/pkg/net/http/pprof/).
GitLab Workhorse can listen on Redis events (currently only builds/register
for runners). This requires you to pass a valid TOML configuration file via
`-config` flag.
For regular setups it only requires the following (replacing the string
with the actual socket)
## Redis
GitLab Workhorse integrates with Redis to do long polling for CI build
requests. This is configured via two things:
- Redis settings in the TOML configuration file
- The `-apiCiLongPollingDuration` command line flag to control polling
behavior for CI build requests
It is OK to enable Redis in the configuration file but to leave CI polling
disabled; this just results in an idle Redis pubsub connection. The
opposite is not possible: CI long polling requires a correct Redis
configuration.
Below we discuss the options for the `[redis]` section in the configuration
file.
```plaintext
[redis]
URL = "unix:///var/run/gitlab/redis.sock"
Password = "my_awesome_password"
Sentinel = [ "tcp://sentinel1:23456", "tcp://sentinel2:23456" ]
SentinelMaster = "mymaster"
```
- `URL` takes a string in the format `unix://path/to/redis.sock` or
`tcp://host:port`.
- `Password` is only required if your Redis instance is password-protected
- `Sentinel` is used if you are using Sentinel.
NOTE:
If both `Sentinel` and `URL` are given, only `Sentinel` will be used.
Optional fields are as follows:
```plaintext
[redis]
DB = 0
MaxIdle = 1
MaxActive = 1
```
- `DB` is the Database to connect to. Defaults to `0`
- `MaxIdle` is how many idle connections can be in the Redis pool at once. Defaults to 1
- `MaxActive` is how many connections the pool can keep. Defaults to 1
## Relative URL support
If you are mounting GitLab at a relative URL, e.g.
`example.com/gitlab`, then you should also use this relative URL in
the `authBackend` setting:
```plaintext
gitlab-workhorse -authBackend http://localhost:8080/gitlab
```
## Interaction of authBackend and authSocket
The interaction between `authBackend` and `authSocket` can be a bit
confusing. It comes down to: if `authSocket` is set it overrides the
_host_ part of `authBackend` but not the relative path.
In table form:
|authBackend|authSocket|Workhorse connects to?|Rails relative URL|
|---|---|---|---|
|unset|unset|`localhost:8080`|`/`|
|`http://localhost:3000`|unset|`localhost:3000`|`/`|
|`http://localhost:3000/gitlab`|unset|`localhost:3000`|`/gitlab`|
|unset|`/path/to/socket`|`/path/to/socket`|`/`|
|`http://localhost:3000`|`/path/to/socket`|`/path/to/socket`|`/`|
|`http://localhost:3000/gitlab`|`/path/to/socket`|`/path/to/socket`|`/gitlab`|
The same applies to `cableBackend` and `cableSocket`.
## Error tracking
GitLab-Workhorse supports remote error tracking with
[Sentry](https://sentry.io). To enable this feature set the
`GITLAB_WORKHORSE_SENTRY_DSN` environment variable.
You can also set the `GITLAB_WORKHORSE_SENTRY_ENVIRONMENT` environment variable to
use the Sentry environment functionality to separate staging, production and
development.
Omnibus (`/etc/gitlab/gitlab.rb`):
```ruby
gitlab_workhorse['env'] = {
'GITLAB_WORKHORSE_SENTRY_DSN' => 'https://foobar'
'GITLAB_WORKHORSE_SENTRY_ENVIRONMENT' => 'production'
}
```
Source installations (`/etc/default/gitlab`):
```plaintext
export GITLAB_WORKHORSE_SENTRY_DSN='https://foobar'
export GITLAB_WORKHORSE_SENTRY_ENVIRONMENT='production'
```
## Distributed Tracing
Workhorse supports distributed tracing through [LabKit](https://gitlab.com/gitlab-org/labkit/) using [OpenTracing APIs](https://opentracing.io).
By default, no tracing implementation is linked into the binary, but different OpenTracing providers can be linked in using [build tags](https://golang.org/pkg/go/build/#hdr-Build_Constraints) or build constraints. This can be done by setting the `BUILD_TAGS` make variable.
For more details of the supported providers, see LabKit, but as an example, for Jaeger tracing support, include the tags: `BUILD_TAGS="tracer_static tracer_static_jaeger"`.
```shell
make BUILD_TAGS="tracer_static tracer_static_jaeger"
```
Once Workhorse is compiled with an opentracing provider, the tracing configuration is configured via the `GITLAB_TRACING` environment variable.
For example:
```shell
GITLAB_TRACING=opentracing://jaeger ./gitlab-workhorse
```
## Continuous Profiling
Workhorse supports continuous profiling through [LabKit](https://gitlab.com/gitlab-org/labkit/) using [Stackdriver Profiler](https://cloud.google.com/profiler).
By default, the Stackdriver Profiler implementation is linked in the binary using [build tags](https://golang.org/pkg/go/build/#hdr-Build_Constraints), though it's not
required and can be skipped.
For example:
```shell
make BUILD_TAGS=""
```
Once Workhorse is compiled with Continuous Profiling, the profiler configuration can be set via `GITLAB_CONTINUOUS_PROFILING`
environment variable.
For example:
```shell
GITLAB_CONTINUOUS_PROFILING="stackdriver?service=workhorse&service_version=1.0.1&project_id=test-123 ./gitlab-workhorse"
```
More information about see the [LabKit monitoring documentation](https://gitlab.com/gitlab-org/labkit/-/blob/master/monitoring/doc.go).
doc/development/workhorse/gitlab_features.md 0 → 100644
View file @ 3165509c
---
stage: Create
group: Source Code
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments
---
# Features that rely on Workhorse
Workhorse itself is not a feature, but there are several features in
GitLab that would not work efficiently without Workhorse.
To put the efficiency benefit in context, consider that in 2020Q3 on
GitLab.com [we see][https://thanos-query.ops.gitlab.net/graph?g0.range_input=1h&g0.max_source_resolution=0s&g0.expr=sum(ruby_process_resident_memory_bytes%7Bapp%3D%22webservice%22%2Cenv%3D%22gprd%22%2Crelease%3D%22gitlab%22%7D)%20%2F%20sum(puma_max_threads%7Bapp%3D%22webservice%22%2Cenv%3D%22gprd%22%2Crelease%3D%22gitlab%22%7D)&g0.tab=1&g1.range_input=1h&g1.max_source_resolution=0s&g1.expr=sum(go_memstats_sys_bytes%7Bapp%3D%22webservice%22%2Cenv%3D%22gprd%22%2Crelease%3D%22gitlab%22%7D)%2Fsum(go_goroutines%7Bapp%3D%22webservice%22%2Cenv%3D%22gprd%22%2Crelease%3D%22gitlab%22%7D)&g1.tab=1]
Rails application threads using on average
about 200MB of RSS vs about 200KB for Workhorse goroutines.
Examples of features that rely on Workhorse:
## 1. `git clone` and `git push` over HTTP
Git clone, pull and push are slow because they transfer large amounts
of data and because each is CPU intensive on the GitLab side. Without
Workhorse, HTTP access to Git repositories would compete with regular
web access to the application, requiring us to run way more Rails
application servers.
## 2. CI runner long polling
GitLab CI runners fetch new CI jobs by polling the GitLab server.
Workhorse acts as a kind of "waiting room" where CI runners can sit
and wait for new CI jobs. Because of Go's efficiency we can fit a lot
of runners in the waiting room at little cost. Without this waiting
room mechanism we would have to add a lot more Rails server capacity.
## 3. File uploads and downloads
File uploads and downloads may be slow either because the file is
large or because the user's connection is slow. Workhorse can handle
the slow part for Rails. This improves the efficiency of features such
as CI artifacts, package repositories, LFS objects, etc.
## 4. Websocket proxying
Features such as the web terminal require a long lived connection
between the user's web browser and a container inside GitLab that is
not directly accessible from the internet. Dedicating a Rails
application thread to proxying such a connection would cost much more
memory than it costs to have Workhorse look after it.
## Quick facts (how does Workhorse work)
- Workhorse can handle some requests without involving Rails at all:
for example, JavaScript files and CSS files are served straight
from disk.
- Workhorse can modify responses sent by Rails: for example if you use
`send_file` in Rails then GitLab Workhorse will open the file on
disk and send its contents as the response body to the client.
- Workhorse can take over requests after asking permission from Rails.
Example: handling `git clone`.
- Workhorse can modify requests before passing them to Rails. Example:
when handling a Git LFS upload Workhorse first asks permission from
Rails, then it stores the request body in a tempfile, then it sends
a modified request containing the tempfile path to Rails.
- Workhorse can manage long-lived WebSocket connections for Rails.
Example: handling the terminal websocket for environments.
- Workhorse does not connect to PostgreSQL, only to Rails and (optionally) Redis.
- We assume that all requests that reach Workhorse pass through an
upstream proxy such as NGINX or Apache first.
- Workhorse does not accept HTTPS connections.
- Workhorse does not clean up idle client connections.
- We assume that all requests to Rails pass through Workhorse.
For more information see ['A brief history of GitLab Workhorse'](https://about.gitlab.com/2016/04/12/a-brief-history-of-gitlab-workhorse/).
doc/development/workhorse/index.md 0 → 100644
View file @ 3165509c
---
stage: Create
group: Source Code
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments
---
# GitLab Workhorse
GitLab Workhorse is a smart reverse proxy for GitLab. It handles
"large" HTTP requests such as file downloads, file uploads, Git
push/pull and Git archive downloads.
Workhorse itself is not a feature, but there are [several features in
GitLab](gitlab_features.md) that would not work efficiently without Workhorse.
The canonical source for Workhorse is
[`gitlab-org/gitlab/workhorse`](https://gitlab.com/gitlab-org/gitlab/tree/master/workhorse).
Prior to [epic #4826](https://gitlab.com/groups/gitlab-org/-/epics/4826), it was
[`gitlab-org/gitlab-workhorse`](https://gitlab.com/gitlab-org/gitlab-workhorse/tree/master),
but that repository is no longer used for development.
## Install Workhorse
To install GitLab Workhorse you need [Go 1.15 or newer](https://golang.org/dl) and
[GNU Make](https://www.gnu.org/software/make/).
To install into `/usr/local/bin` run `make install`.
```plaintext
make install
```
To install into `/foo/bin` set the PREFIX variable.
```plaintext
make install PREFIX=/foo
```
On some operating systems, such as FreeBSD, you may have to use
`gmake` instead of `make`.
*NOTE*: Some features depends on build tags, make sure to check
[Workhorse configuration](configuration.md) to enable them.
### Run time dependencies
Workhorse uses [Exiftool](https://www.sno.phy.queensu.ca/~phil/exiftool/) for
removing EXIF data (which may contain sensitive information) from uploaded
images. If you installed GitLab:
- Using the Omnibus package, you're all set.
*NOTE* that if you are using CentOS Minimal, you may need to install `perl`
package: `yum install perl`
- From source, make sure `exiftool` is installed:
```shell
# Debian/Ubuntu
sudo apt-get install libimage-exiftool-perl
# RHEL/CentOS
sudo yum install perl-Image-ExifTool
```
## Testing your code
Run the tests with:
```plaintext
make clean test
```
Each feature in GitLab Workhorse should have an integration test that
verifies that the feature 'kicks in' on the right requests and leaves
other requests unaffected. It is better to also have package-level tests
for specific behavior but the high-level integration tests should have
the first priority during development.
It is OK if a feature is only covered by integration tests.
<!--
## License
This code is distributed under the MIT license, see the [LICENSE](LICENSE) file.
-->
doc/development/workhorse/new_features.md 0 → 100644
View file @ 3165509c
---
stage: Create
group: Source Code
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments
---
# Adding new features to Workhorse
GitLab Workhorse is a smart reverse proxy for GitLab. It handles
"long" HTTP requests such as file downloads, file uploads, Git
push/pull and Git archive downloads.
Workhorse itself is not a feature, but there are [several features in GitLab](gitlab_features.md) that would not work efficiently without Workhorse.
At a first glance, it may look like Workhorse is just a pipeline for processing HTTP streams so that you can reduce the amount of logic in your Ruby on Rails controller, but there are good reasons to avoid treating it like that.
Engineers embarking on the quest of offloading a feature to Workhorse often find that the endeavor is much higher than what originally anticipated. In part because of the new programming language (only a few engineers at GitLab are Go developers), in part because of the demanding requirements for Workhorse. Workhorse is stateless, memory and disk usage must be kept under tight control, and the request should not be slowed down in the process.
## Can I add a new feature to Workhorse?
We suggest to follow this route only if absolutely necessary and no other options are available.
Splitting a feature between the Rails code-base and Workhorse is deliberately choosing to introduce technical debt. It adds complexity to the system and coupling between the two components.
- Building features using Workhorse has a considerable complexity cost, so you should prefer designs based on Rails requests and Sidekiq jobs.
- Even when using Rails+Sidekiq is "more work" than using Rails+Workhorse, Rails+Sidekiq is easier to maintain in the long term because Workhorse is unique to GitLab while Rails+Sidekiq is an industry standard.
- For "global" behaviors around web requests consider using a Rack middleware instead of Workhorse.
- Generally speaking, we should only use Rails+Workhorse if the HTTP client expects behavior that is not reasonable to implement in Rails, like "long" requests.
## What is a "long" request?
There is one order of magnitude between Workhorse and Puma RAM usage. Having connection open for a period longer than milliseconds is a problem because of the amount of RAM it monopolizes once it reaches the Ruby on Rails controller.
So far we identified two classes of "long" requests: data transfers and HTTP long polling.
`git push`, `git pull`, uploading or downloading an artifact, the CI runner waiting for a new job are all good examples of long requests.
With the rise of cloud-native installations, Workhorse's feature-set was extended to add object storage direct-upload, to get rid of the shared Network File System (NFS) drives.
In 2020 @nolith presented at FOSDEM a talk titled [_Speed up the monolith. Building a smart reverse proxy in Go_](https://archive.fosdem.org/2020/schedule/event/speedupmonolith/).
You can watch the recording for more details on the history of Workhorse and the NFS removal.
[Uploads development documentation](../uploads.md)
contains the most common use-cases for adding a new type of upload and may answer all of your questions.
If you still think we should add a new feature to Workhorse, please open an issue explaining **what you want to implement** and **why it can't be implemented in our Ruby code-base**. Workhorse maintainers will be happy to help you assessing the situation.
workhorse/README.md
View file @ 3165509c
......@@ -4,37 +4,11 @@ group: Source Code
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments
---
# GitLab Workhorse
# Workhorse documentation
GitLab Workhorse is a smart reverse proxy for GitLab. It handles
"large" HTTP requests such as file downloads, file uploads, Git
push/pull and Git archive downloads.
This document was moved to [another location](../doc/development/workhorse/index.md).
Workhorse itself is not a feature, but there are [several features in
GitLab](doc/architecture/gitlab_features.md) that would not work efficiently without Workhorse.
## Canonical source
The canonical source for Workhorse is
[`gitlab-org/gitlab/workhorse`](https://gitlab.com/gitlab-org/gitlab/tree/master/workhorse).
Prior to [epic #4826](https://gitlab.com/groups/gitlab-org/-/epics/4826), it was
[`gitlab-org/gitlab-workhorse`](https://gitlab.com/gitlab-org/gitlab-workhorse/tree/master),
but that repository is no longer used for development.
## Documentation
Workhorse documentation is available in the [`doc` folder of this repository](doc/).
- Architectural overview
- [GitLab features that rely on Workhorse](doc/architecture/gitlab_features.md)
- [Websocket channel support](doc/architecture/channel.md)
- Operating Workhorse
- [Source installation](doc/operations/install.md)
- [Workhorse configuration](doc/operations/configuration.md)
- [Contributing](CONTRIBUTING.md)
- [Adding new features](doc/development/new_features.md)
- [Testing your code](doc/development/tests.md)
## License
This code is distributed under the MIT license, see the [LICENSE](LICENSE) file.
<!-- This redirect file can be deleted after <2022-07-01>. -->
<!-- Redirects that point to other docs in the same project expire in three months. -->
<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. -->
<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html -->
workhorse/doc/architecture/channel.md
View file @ 3165509c
......@@ -2,198 +2,15 @@
stage: Create
group: Source Code
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments
redirect_to: '../../../doc/development/backend/create_source_code_be/workhorse/channel.md'
remove_date: '2022-07-01'
---
# Websocket channel support
In some cases, GitLab can provide in-browser terminal access to an
environment (which is a running server or container, onto which a
project has been deployed), or even access to services running in CI
through a WebSocket. Workhorse manages the WebSocket upgrade and
long-lived connection to the websocket connection, which frees
up GitLab to process other requests.
This document was moved to [another location](../../../doc/development/workhorse/channel.md).
This document outlines the architecture of these connections.
## Introduction to WebSockets
A websocket is an "upgraded" HTTP/1.1 request. Their purpose is to
permit bidirectional communication between a client and a server.
**Websockets are not HTTP**. Clients can send messages (known as
frames) to the server at any time, and vice-versa. Client messages
are not necessarily requests, and server messages are not necessarily
responses. WebSocket URLs have schemes like `ws://` (unencrypted) or
`wss://` (TLS-secured).
When requesting an upgrade to WebSocket, the browser sends a HTTP/1.1
request that looks like this:
```plaintext
GET /path.ws HTTP/1.1
Connection: upgrade
Upgrade: websocket
Sec-WebSocket-Protocol: terminal.gitlab.com
# More headers, including security measures
```
At this point, the connection is still HTTP, so this is a request and
the server can send a normal HTTP response, including `404 Not Found`,
`500 Internal Server Error`, etc.
If the server decides to permit the upgrade, it will send a HTTP
`101 Switching Protocols` response. From this point, the connection
is no longer HTTP. It is a WebSocket and frames, not HTTP requests,
will flow over it. The connection will persist until the client or
server closes the connection.
In addition to the subprotocol, individual websocket frames may
also specify a message type - examples include `BinaryMessage`,
`TextMessage`, `Ping`, `Pong` or `Close`. Only binary frames can
contain arbitrary data - other frames are expected to be valid
UTF-8 strings, in addition to any subprotocol expectations.
## Browser to Workhorse
Using the terminal as an example, GitLab serves a JavaScript terminal
emulator to the browser on a URL like
`https://gitlab.com/group/project/-/environments/1/terminal`.
This opens a websocket connection to, e.g.,
`wss://gitlab.com/group/project/-/environments/1/terminal.ws`,
This endpoint doesn't exist in GitLab - only in Workhorse.
When receiving the connection, Workhorse first checks that the
client is authorized to access the requested terminal. It does
this by performing a "preauthentication" request to GitLab.
If the client has the appropriate permissions and the terminal
exists, GitLab responds with a successful response that includes
details of the terminal that the client should be connected to.
Otherwise, it returns an appropriate HTTP error response.
Errors are passed back to the client as HTTP responses, but if
GitLab returns valid terminal details to Workhorse, it will
connect to the specified terminal, upgrade the browser to a
WebSocket, and proxy between the two connections for as long
as the browser's credentials are valid. Workhorse will also
send regular `PingMessage` control frames to the browser, to
keep intervening proxies from terminating the connection
while the browser is present.
The browser must request an upgrade with a specific subprotocol:
### `terminal.gitlab.com`
This subprotocol considers `TextMessage` frames to be invalid.
Control frames, such as `PingMessage` or `CloseMessage`, have
their usual meanings.
`BinaryMessage` frames sent from the browser to the server are
arbitrary text input.
`BinaryMessage` frames sent from the server to the browser are
arbitrary text output.
These frames are expected to contain ANSI text control codes
and may be in any encoding.
### `base64.terminal.gitlab.com`
This subprotocol considers `BinaryMessage` frames to be invalid.
Control frames, such as `PingMessage` or `CloseMessage`, have
their usual meanings.
`TextMessage` frames sent from the browser to the server are
base64-encoded arbitrary text input (so the server must
base64-decode them before inputting them).
`TextMessage` frames sent from the server to the browser are
base64-encoded arbitrary text output (so the browser must
base64-decode them before outputting them).
In their base64-encoded form, these frames are expected to
contain ANSI terminal control codes, and may be in any encoding.
## Workhorse to GitLab
Using again the terminal as an example, before upgrading the browser,
Workhorse sends a normal HTTP request to GitLab on a URL like
`https://gitlab.com/group/project/environments/1/terminal.ws/authorize`.
This returns a JSON response containing details of where the
terminal can be found, and how to connect it. In particular,
the following details are returned in case of success:
- WebSocket URL to **connect** to, e.g.: `wss://example.com/terminals/1.ws?tty=1`
- WebSocket subprotocols to support, e.g.: `["channel.k8s.io"]`
- Headers to send, e.g.: `Authorization: Token xxyyz..`
- Certificate authority to verify `wss` connections with (optional)
Workhorse periodically re-checks this endpoint, and if it gets an
error response, or the details of the terminal change, it will
terminate the websocket session.
## Workhorse to the WebSocket server
In GitLab, environments or CI jobs may have a deployment service (e.g.,
`KubernetesService`) associated with them. This service knows
where the terminals or the service for an environment may be found, and these
details are returned to Workhorse by GitLab.
These URLs are *also* WebSocket URLs, and GitLab tells Workhorse
which subprotocols to speak over the connection, along with any
authentication details required by the remote end.
Before upgrading the browser's connection to a websocket,
Workhorse opens a HTTP client connection, according to the
details given to it by Workhorse, and attempts to upgrade
that connection to a websocket. If it fails, an error
response is sent to the browser; otherwise, the browser is
also upgraded.
Workhorse now has two websocket connections, albeit with
differing subprotocols. It decodes incoming frames from the
browser, re-encodes them to the channel's subprotocol, and
sends them to the channel. Similarly, it decodes incoming
frames from the channel, re-encodes them to the browser's
subprotocol, and sends them to the browser.
When either connection closes or enters an error state,
Workhorse detects the error and closes the other connection,
terminating the channel session. If the browser is the
connection that has disconnected, Workhorse will send an ANSI
`End of Transmission` control code (the `0x04` byte) to the
channel, encoded according to the appropriate subprotocol.
Workhorse will automatically reply to any websocket ping frame
sent by the channel, to avoid being disconnected.
Currently, Workhorse only supports the following subprotocols.
Supporting new deployment services will require new subprotocols
to be supported:
### `channel.k8s.io`
Used by Kubernetes, this subprotocol defines a simple multiplexed
channel.
Control frames have their usual meanings. `TextMessage` frames are
invalid. `BinaryMessage` frames represent I/O to a specific file
descriptor.
The first byte of each `BinaryMessage` frame represents the file
descriptor (fd) number, as a `uint8` (so the value `0x00` corresponds
to fd 0, `STDIN`, while `0x01` corresponds to fd 1, `STDOUT`).
The remaining bytes represent arbitrary data. For frames received
from the server, they are bytes that have been received from that
fd. For frames sent to the server, they are bytes that should be
written to that fd.
### `base64.channel.k8s.io`
Also used by Kubernetes, this subprotocol defines a similar multiplexed
channel to `channel.k8s.io`. The main differences are:
- `TextMessage` frames are valid, rather than `BinaryMessage` frames.
- The first byte of each `TextMessage` frame represents the file
descriptor as a numeric UTF-8 character, so the character `U+0030`,
or "0", is fd 0, STDIN).
- The remaining bytes represent base64-encoded arbitrary data.
<!-- This redirect file can be deleted after <2022-07-01>. -->
<!-- Redirects that point to other docs in the same project expire in three months. -->
<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. -->
<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html -->
workhorse/doc/architecture/gitlab_features.md
View file @ 3165509c
......@@ -6,68 +6,9 @@ info: To determine the technical writer assigned to the Stage/Group associated w
# Features that rely on Workhorse
Workhorse itself is not a feature, but there are several features in
GitLab that would not work efficiently without Workhorse.
This document was moved to [another location](../../../doc/development/workhorse/gitlab_features.md).
To put the efficiency benefit in context, consider that in 2020Q3 on
GitLab.com [we see][https://thanos-query.ops.gitlab.net/graph?g0.range_input=1h&g0.max_source_resolution=0s&g0.expr=sum(ruby_process_resident_memory_bytes%7Bapp%3D%22webservice%22%2Cenv%3D%22gprd%22%2Crelease%3D%22gitlab%22%7D)%20%2F%20sum(puma_max_threads%7Bapp%3D%22webservice%22%2Cenv%3D%22gprd%22%2Crelease%3D%22gitlab%22%7D)&g0.tab=1&g1.range_input=1h&g1.max_source_resolution=0s&g1.expr=sum(go_memstats_sys_bytes%7Bapp%3D%22webservice%22%2Cenv%3D%22gprd%22%2Crelease%3D%22gitlab%22%7D)%2Fsum(go_goroutines%7Bapp%3D%22webservice%22%2Cenv%3D%22gprd%22%2Crelease%3D%22gitlab%22%7D)&g1.tab=1]
Rails application threads using on average
about 200MB of RSS vs about 200KB for Workhorse goroutines.
Examples of features that rely on Workhorse:
## 1. `git clone` and `git push` over HTTP
Git clone, pull and push are slow because they transfer large amounts
of data and because each is CPU intensive on the GitLab side. Without
Workhorse, HTTP access to Git repositories would compete with regular
web access to the application, requiring us to run way more Rails
application servers.
## 2. CI runner long polling
GitLab CI runners fetch new CI jobs by polling the GitLab server.
Workhorse acts as a kind of "waiting room" where CI runners can sit
and wait for new CI jobs. Because of Go's efficiency we can fit a lot
of runners in the waiting room at little cost. Without this waiting
room mechanism we would have to add a lot more Rails server capacity.
## 3. File uploads and downloads
File uploads and downloads may be slow either because the file is
large or because the user's connection is slow. Workhorse can handle
the slow part for Rails. This improves the efficiency of features such
as CI artifacts, package repositories, LFS objects, etc.
## 4. Websocket proxying
Features such as the web terminal require a long lived connection
between the user's web browser and a container inside GitLab that is
not directly accessible from the internet. Dedicating a Rails
application thread to proxying such a connection would cost much more
memory than it costs to have Workhorse look after it.
## Quick facts (how does Workhorse work)
- Workhorse can handle some requests without involving Rails at all:
for example, JavaScript files and CSS files are served straight
from disk.
- Workhorse can modify responses sent by Rails: for example if you use
`send_file` in Rails then GitLab Workhorse will open the file on
disk and send its contents as the response body to the client.
- Workhorse can take over requests after asking permission from Rails.
Example: handling `git clone`.
- Workhorse can modify requests before passing them to Rails. Example:
when handling a Git LFS upload Workhorse first asks permission from
Rails, then it stores the request body in a tempfile, then it sends
a modified request containing the tempfile path to Rails.
- Workhorse can manage long-lived WebSocket connections for Rails.
Example: handling the terminal websocket for environments.
- Workhorse does not connect to PostgreSQL, only to Rails and (optionally) Redis.
- We assume that all requests that reach Workhorse pass through an
upstream proxy such as NGINX or Apache first.
- Workhorse does not accept HTTPS connections.
- Workhorse does not clean up idle client connections.
- We assume that all requests to Rails pass through Workhorse.
For more information see ['A brief history of GitLab Workhorse'](https://about.gitlab.com/2016/04/12/a-brief-history-of-gitlab-workhorse/).
<!-- This redirect file can be deleted after <2022-07-01>. -->
<!-- Redirects that point to other docs in the same project expire in three months. -->
<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. -->
<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html -->
workhorse/doc/channel.md
View file @ 3165509c
This file was moved to [`architecture/channel.md`](doc/architecture/channel.md).
---
stage: Create
group: Source Code
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments
---
# Websocket channel support
This document was moved to [another location](../../doc/development/workhorse/channel.md).
<!-- This redirect file can be deleted after <2022-07-01>. -->
<!-- Redirects that point to other docs in the same project expire in three months. -->
<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. -->
<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html -->
workhorse/doc/development/new_features.md
View file @ 3165509c
......@@ -6,41 +6,9 @@ info: To determine the technical writer assigned to the Stage/Group associated w
# Adding new features to Workhorse
GitLab Workhorse is a smart reverse proxy for GitLab. It handles
"long" HTTP requests such as file downloads, file uploads, Git
push/pull and Git archive downloads.
This document was moved to [another location](../../../doc/development/workhorse/new_features.md).
Workhorse itself is not a feature, but there are [several features in GitLab](https://gitlab.com/gitlab-org/gitlab/-/blob/master/workhorse/doc/architecture/gitlab_features.md) that would not work efficiently without Workhorse.
At a first glance, it may look like Workhorse is just a pipeline for processing HTTP streams so that you can reduce the amount of logic in your Ruby on Rails controller, but there are good reasons to avoid treating it like that.
Engineers embarking on the quest of offloading a feature to Workhorse often find that the endeavor is much higher than what originally anticipated. In part because of the new programming language (only a few engineers at GitLab are Go developers), in part because of the demanding requirements for Workhorse. Workhorse is stateless, memory and disk usage must be kept under tight control, and the request should not be slowed down in the process.
## Can I add a new feature to Workhorse?
We suggest to follow this route only if absolutely necessary and no other options are available.
Splitting a feature between the Rails code-base and Workhorse is deliberately choosing to introduce technical debt. It adds complexity to the system and coupling between the two components.
- Building features using Workhorse has a considerable complexity cost, so you should prefer designs based on Rails requests and Sidekiq jobs.
- Even when using Rails+Sidekiq is "more work" than using Rails+Workhorse, Rails+Sidekiq is easier to maintain in the long term because Workhorse is unique to GitLab while Rails+Sidekiq is an industry standard.
- For "global" behaviors around web requests consider using a Rack middleware instead of Workhorse.
- Generally speaking, we should only use Rails+Workhorse if the HTTP client expects behavior that is not reasonable to implement in Rails, like "long" requests.
## What is a "long" request?
There is one order of magnitude between Workhorse and Puma RAM usage. Having connection open for a period longer than milliseconds is a problem because of the amount of RAM it monopolizes once it reaches the Ruby on Rails controller.
So far we identified two classes of "long" requests: data transfers and HTTP long polling.
`git push`, `git pull`, uploading or downloading an artifact, the CI runner waiting for a new job are all good examples of long requests.
With the rise of cloud-native installations, Workhorse's feature-set was extended to add object storage direct-upload, to get rid of the shared Network File System (NFS) drives.
In 2020 @nolith presented at FOSDEM a talk titled [_Speed up the monolith. Building a smart reverse proxy in Go_](https://archive.fosdem.org/2020/schedule/event/speedupmonolith/).
You can watch the recording for more details on the history of Workhorse and the NFS removal.
[Uploads development documentation]( https://docs.gitlab.com/ee/development/uploads.html)
contains the most common use-cases for adding a new type of upload and may answer all of your questions.
If you still think we should add a new feature to Workhorse, please open an issue explaining **what you want to implement** and **why it can't be implemented in our Ruby code-base**. Workhorse maintainers will be happy to help you assessing the situation.
<!-- This redirect file can be deleted after <2022-07-01>. -->
<!-- Redirects that point to other docs in the same project expire in three months. -->
<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. -->
<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html -->
workhorse/doc/development/tests.md
View file @ 3165509c
......@@ -6,18 +6,9 @@ info: To determine the technical writer assigned to the Stage/Group associated w
# Testing your code
Run the tests with:
This document was moved to [another location](../../../doc/development/workhorse/index.md).
```plaintext
make clean test
```
## Coverage / what to test
Each feature in GitLab Workhorse should have an integration test that
verifies that the feature 'kicks in' on the right requests and leaves
other requests unaffected. It is better to also have package-level tests
for specific behavior but the high-level integration tests should have
the first priority during development.
It is OK if a feature is only covered by integration tests.
<!-- This redirect file can be deleted after <2022-07-01>. -->
<!-- Redirects that point to other docs in the same project expire in three months. -->
<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. -->
<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html -->
workhorse/doc/operations/configuration.md
View file @ 3165509c
......@@ -6,214 +6,9 @@ info: To determine the technical writer assigned to the Stage/Group associated w
# Workhorse configuration
For historical reasons Workhorse uses both command line flags, a configuration file and environment variables.
This document was moved to [another location](../../../doc/development/workhorse/configuration.md).
All new configuration options that get added to Workhorse should go into the configuration file.
## CLI options
```plaintext
gitlab-workhorse [OPTIONS]
Options:
-apiCiLongPollingDuration duration
Long polling duration for job requesting for runners (default 50ns)
-apiLimit uint
Number of API requests allowed at single time
-apiQueueDuration duration
Maximum queueing duration of requests (default 30s)
-apiQueueLimit uint
Number of API requests allowed to be queued
-authBackend string
Authentication/authorization backend (default "http://localhost:8080")
-authSocket string
Optional: Unix domain socket to dial authBackend at
-cableBackend string
Optional: ActionCable backend (default authBackend)
-cableSocket string
Optional: Unix domain socket to dial cableBackend at (default authSocket)
-config string
TOML file to load config from
-developmentMode
Allow the assets to be served from Rails app
-documentRoot string
Path to static files content (default "public")
-listenAddr string
Listen address for HTTP server (default "localhost:8181")
-listenNetwork string
Listen 'network' (tcp, tcp4, tcp6, unix) (default "tcp")
-listenUmask int
Umask for Unix socket
-logFile string
Log file location
-logFormat string
Log format to use defaults to text (text, json, structured, none) (default "text")
-pprofListenAddr string
pprof listening address, e.g. 'localhost:6060'
-prometheusListenAddr string
Prometheus listening address, e.g. 'localhost:9229'
-proxyHeadersTimeout duration
How long to wait for response headers when proxying the request (default 5m0s)
-secretPath string
File with secret key to authenticate with authBackend (default "./.gitlab_workhorse_secret")
-version
Print version and exit
```
The 'auth backend' refers to the GitLab Rails application. The name is
a holdover from when GitLab Workhorse only handled Git push/pull over
HTTP.
GitLab Workhorse can listen on either a TCP or a Unix domain socket. It
can also open a second listening TCP listening socket with the Go
[`net/http/pprof` profiler server](http://golang.org/pkg/net/http/pprof/).
GitLab Workhorse can listen on Redis events (currently only builds/register
for runners). This requires you to pass a valid TOML configuration file via
`-config` flag.
For regular setups it only requires the following (replacing the string
with the actual socket)
## Redis
GitLab Workhorse integrates with Redis to do long polling for CI build
requests. This is configured via two things:
- Redis settings in the TOML configuration file
- The `-apiCiLongPollingDuration` command line flag to control polling
behavior for CI build requests
It is OK to enable Redis in the configuration file but to leave CI polling
disabled; this just results in an idle Redis pubsub connection. The
opposite is not possible: CI long polling requires a correct Redis
configuration.
Below we discuss the options for the `[redis]` section in the configuration
file.
```plaintext
[redis]
URL = "unix:///var/run/gitlab/redis.sock"
Password = "my_awesome_password"
Sentinel = [ "tcp://sentinel1:23456", "tcp://sentinel2:23456" ]
SentinelMaster = "mymaster"
```
- `URL` takes a string in the format `unix://path/to/redis.sock` or
`tcp://host:port`.
- `Password` is only required if your Redis instance is password-protected
- `Sentinel` is used if you are using Sentinel.
NOTE:
If both `Sentinel` and `URL` are given, only `Sentinel` will be used.
Optional fields are as follows:
```plaintext
[redis]
DB = 0
MaxIdle = 1
MaxActive = 1
```
- `DB` is the Database to connect to. Defaults to `0`
- `MaxIdle` is how many idle connections can be in the Redis pool at once. Defaults to 1
- `MaxActive` is how many connections the pool can keep. Defaults to 1
## Relative URL support
If you are mounting GitLab at a relative URL, e.g.
`example.com/gitlab`, then you should also use this relative URL in
the `authBackend` setting:
```plaintext
gitlab-workhorse -authBackend http://localhost:8080/gitlab
```
## Interaction of authBackend and authSocket
The interaction between `authBackend` and `authSocket` can be a bit
confusing. It comes down to: if `authSocket` is set it overrides the
_host_ part of `authBackend` but not the relative path.
In table form:
|authBackend|authSocket|Workhorse connects to?|Rails relative URL|
|---|---|---|---|
|unset|unset|`localhost:8080`|`/`|
|`http://localhost:3000`|unset|`localhost:3000`|`/`|
|`http://localhost:3000/gitlab`|unset|`localhost:3000`|`/gitlab`|
|unset|`/path/to/socket`|`/path/to/socket`|`/`|
|`http://localhost:3000`|`/path/to/socket`|`/path/to/socket`|`/`|
|`http://localhost:3000/gitlab`|`/path/to/socket`|`/path/to/socket`|`/gitlab`|
The same applies to `cableBackend` and `cableSocket`.
## Error tracking
GitLab-Workhorse supports remote error tracking with
[Sentry](https://sentry.io). To enable this feature set the
`GITLAB_WORKHORSE_SENTRY_DSN` environment variable.
You can also set the `GITLAB_WORKHORSE_SENTRY_ENVIRONMENT` environment variable to
use the Sentry environment functionality to separate staging, production and
development.
Omnibus (`/etc/gitlab/gitlab.rb`):
```ruby
gitlab_workhorse['env'] = {
'GITLAB_WORKHORSE_SENTRY_DSN' => 'https://foobar'
'GITLAB_WORKHORSE_SENTRY_ENVIRONMENT' => 'production'
}
```
Source installations (`/etc/default/gitlab`):
```plaintext
export GITLAB_WORKHORSE_SENTRY_DSN='https://foobar'
export GITLAB_WORKHORSE_SENTRY_ENVIRONMENT='production'
```
## Distributed Tracing
Workhorse supports distributed tracing through [LabKit](https://gitlab.com/gitlab-org/labkit/) using [OpenTracing APIs](https://opentracing.io).
By default, no tracing implementation is linked into the binary, but different OpenTracing providers can be linked in using [build tags](https://golang.org/pkg/go/build/#hdr-Build_Constraints) or build constraints. This can be done by setting the `BUILD_TAGS` make variable.
For more details of the supported providers, see LabKit, but as an example, for Jaeger tracing support, include the tags: `BUILD_TAGS="tracer_static tracer_static_jaeger"`.
```shell
make BUILD_TAGS="tracer_static tracer_static_jaeger"
```
Once Workhorse is compiled with an opentracing provider, the tracing configuration is configured via the `GITLAB_TRACING` environment variable.
For example:
```shell
GITLAB_TRACING=opentracing://jaeger ./gitlab-workhorse
```
## Continuous Profiling
Workhorse supports continuous profiling through [LabKit](https://gitlab.com/gitlab-org/labkit/) using [Stackdriver Profiler](https://cloud.google.com/profiler).
By default, the Stackdriver Profiler implementation is linked in the binary using [build tags](https://golang.org/pkg/go/build/#hdr-Build_Constraints), though it's not
required and can be skipped.
For example:
```shell
make BUILD_TAGS=""
```
Once Workhorse is compiled with Continuous Profiling, the profiler configuration can be set via `GITLAB_CONTINUOUS_PROFILING`
environment variable.
For example:
```shell
GITLAB_CONTINUOUS_PROFILING="stackdriver?service=workhorse&service_version=1.0.1&project_id=test-123 ./gitlab-workhorse"
```
More information about see the [LabKit monitoring documentation](https://gitlab.com/gitlab-org/labkit/-/blob/master/monitoring/doc.go).
<!-- This redirect file can be deleted after <2022-07-01>. -->
<!-- Redirects that point to other docs in the same project expire in three months. -->
<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. -->
<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html -->
workhorse/doc/operations/install.md
View file @ 3165509c
# Installation
To install GitLab Workhorse you need [Go 1.15 or
newer](https://golang.org/dl) and [GNU
Make](https://www.gnu.org/software/make/).
To install into `/usr/local/bin` run `make install`.
```plaintext
make install
```
To install into `/foo/bin` set the PREFIX variable.
```plaintext
make install PREFIX=/foo
```
---
stage: Create
group: Source Code
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments
---
On some operating systems, such as FreeBSD, you may have to use
`gmake` instead of `make`.
*NOTE*: Some features depends on build tags, make sure to check
[Workhorse configuration](doc/operations/configuration.md) to enable them.
## Run time dependencies
### Exiftool
Workhorse uses [Exiftool](https://www.sno.phy.queensu.ca/~phil/exiftool/) for
removing EXIF data (which may contain sensitive information) from uploaded
images. If you installed GitLab:
- Using the Omnibus package, you're all set.
*NOTE* that if you are using CentOS Minimal, you may need to install `perl`
package: `yum install perl`
- From source, make sure `exiftool` is installed:
# Installation
```shell
# Debian/Ubuntu
sudo apt-get install libimage-exiftool-perl
This document was moved to [another location](../../../doc/development/workhorse/index.md).
# RHEL/CentOS
sudo yum install perl-Image-ExifTool
```
<!-- This redirect file can be deleted after <2022-07-01>. -->
<!-- Redirects that point to other docs in the same project expire in three months. -->
<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. -->
<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html -->
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment
GitLab Nexedi Edition | About GitLab | About Nexedi | 沪ICP备2021021310号-2 | 沪ICP备2021021310号-7