Skip to content

G
gitlab-ce
Commit b734365b authored by Marcel Amirault's avatar Marcel Amirault
Browse files
Options

Move 6 docs out of the workflow dir 

Move forking_workflow.md, file_finder.md, git_annex.md
and the 3 /lfs docs. Move images as well, and update
all links in the docs, and links to the docs too. Finally,
leave redirects in the old doc locations
parent 0848f550
Hide whitespace changes
Inline Side-by-side
Showing with 1150 additions and 1120 deletions
doc/administration/git_annex.md 0 → 100644
View file @ b734365b
# Git annex
> **Warning:** GitLab has [completely
removed][deprecate-annex-issue] in GitLab 9.0 (2017/03/22).
Read through the [migration guide from git-annex to Git LFS][guide].
The biggest limitation of Git, compared to some older centralized version
control systems, has been the maximum size of the repositories.
The general recommendation is to not have Git repositories larger than 1GB to
preserve performance. Although GitLab has no limit (some repositories in GitLab
are over 50GB!), we subscribe to the advice to keep repositories as small as
you can.
Not being able to version control large binaries is a big problem for many
larger organizations.
Videos, photos, audio, compiled binaries and many other types of files are too
large. As a workaround, people keep artwork-in-progress in a Dropbox folder and
only check in the final result. This results in using outdated files, not
having a complete history and increases the risk of losing work.
This problem is solved in GitLab Enterprise Edition by integrating the
[git-annex] application.
`git-annex` allows managing large binaries with Git without checking the
contents into Git.
You check-in only a symlink that contains the SHA-1 of the large binary. If you
need the large binary, you can sync it from the GitLab server over `rsync`, a
very fast file copying tool.
## GitLab git-annex Configuration
`git-annex` is disabled by default in GitLab. Below you will find the
configuration options required to enable it.
### Requirements
`git-annex` needs to be installed both on the server and the client side.
For Debian-like systems (e.g., Debian, Ubuntu) this can be achieved by running:
```
sudo apt-get update && sudo apt-get install git-annex
```
For RedHat-like systems (e.g., CentOS, RHEL) this can be achieved by running:
```
sudo yum install epel-release && sudo yum install git-annex
```
### Configuration for Omnibus packages
For Omnibus GitLab packages, only one configuration setting is needed.
The Omnibus package will internally set the correct options in all locations.
1. In `/etc/gitlab/gitlab.rb` add the following line:
```ruby
gitlab_shell['git_annex_enabled'] = true
```
1. Save the file and [reconfigure GitLab][] for the changes to take effect.
### Configuration for installations from source
There are 2 settings to enable git-annex on your GitLab server.
One is located in `config/gitlab.yml` of the GitLab repository and the other
one is located in `config.yml` of GitLab Shell.
1. In `config/gitlab.yml` add or edit the following lines:
```yaml
gitlab_shell:
git_annex_enabled: true
```
1. In `config.yml` of GitLab Shell add or edit the following lines:
```yaml
git_annex_enabled: true
```
1. Save the files and [restart GitLab][] for the changes to take effect.
## Using GitLab git-annex
> **Note:**
> Your Git remotes must be using the SSH protocol, not HTTP(S).
Here is an example workflow of uploading a very large file and then checking it
into your Git repository:
```bash
git clone git@example.com:group/project.git
git annex init 'My Laptop' # initialize the annex project and give an optional description
cp ~/tmp/debian.iso ./ # copy a large file into the current directory
git annex add debian.iso # add the large file to git annex
git commit -am "Add Debian iso" # commit the file metadata
git annex sync --content # sync the Git repo and large file to the GitLab server
```
The output should look like this:
```
commit
On branch master
Your branch is ahead of 'origin/master' by 1 commit.
(use "git push" to publish your local commits)
nothing to commit, working tree clean
ok
pull origin
remote: Counting objects: 5, done.
remote: Compressing objects: 100% (4/4), done.
remote: Total 5 (delta 2), reused 0 (delta 0)
Unpacking objects: 100% (5/5), done.
From example.com:group/project
497842b..5162f80 git-annex -> origin/git-annex
ok
(merging origin/git-annex into git-annex...)
(recording state in git...)
copy debian.iso (checking origin...) (to origin...)
SHA256E-s26214400--8092b3d482fb1b7a5cf28c43bc1425c8f2d380e86869c0686c49aa7b0f086ab2.iso
26,214,400 100% 638.88kB/s 0:00:40 (xfr#1, to-chk=0/1)
ok
pull origin
ok
(recording state in git...)
push origin
Counting objects: 15, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (13/13), done.
Writing objects: 100% (15/15), 1.64 KiB | 0 bytes/s, done.
Total 15 (delta 1), reused 0 (delta 0)
To example.com:group/project.git
* [new branch] git-annex -> synced/git-annex
* [new branch] master -> synced/master
ok
```
Your files can be found in the `master` branch, but you'll notice that there
are more branches created by the `annex sync` command.
Git Annex will also create a new directory at `.git/annex/` and will record the
tracked files in the `.git/config` file. The files you assign to be tracked
with `git-annex` will not affect the existing `.git/config` records. The files
are turned into symbolic links that point to data in `.git/annex/objects/`.
The `debian.iso` file in the example will contain the symbolic link:
```
.git/annex/objects/ZW/1k/SHA256E-s82701--6384039733b5035b559efd5a2e25a493ab6e09aabfd5162cc03f6f0ec238429d.png/SHA256E-s82701--6384039733b5035b559efd5a2e25a493ab6e09aabfd5162cc03f6f0ec238429d.iso
```
Use `git annex info` to retrieve the information about the local copy of your
repository.
---
Downloading a single large file is also very simple:
```bash
git clone git@gitlab.example.com:group/project.git
git annex sync # sync Git branches but not the large file
git annex get debian.iso # download the large file
```
To download all files:
```bash
git clone git@gitlab.example.com:group/project.git
git annex sync --content # sync Git branches and download all the large files
```
By using `git-annex` without GitLab, anyone that can access the server can also
access the files of all projects, but GitLab Annex ensures that you can only
access files of projects you have access to (developer, maintainer, or owner role).
## How it works
Internally GitLab uses [GitLab Shell] to handle SSH access and this was a great
integration point for `git-annex`.
There is a setting in GitLab Shell so you can disable GitLab Annex support
if you want to.
## Troubleshooting tips
Differences in version of `git-annex` on the GitLab server and on local machines
can cause `git-annex` to raise unpredicted warnings and errors.
Consult the [Annex upgrade page][annex-upgrade] for more information about
the differences between versions. You can find out which version is installed
on your server by navigating to <https://pkgs.org/download/git-annex> and
searching for your distribution.
Although there is no general guide for `git-annex` errors, there are a few tips
on how to go around the warnings.
### `git-annex-shell: Not a git-annex or gcrypt repository`
This warning can appear on the initial `git annex sync --content` and is caused
by differences in `git-annex-shell`. You can read more about it
[in this git-annex issue][issue].
One important thing to note is that despite the warning, the `sync` succeeds
and the files are pushed to the GitLab repository.
If you get hit by this, you can run the following command inside the repository
that the warning was raised:
```
git config remote.origin.annex-ignore false
```
Consecutive runs of `git annex sync --content` **should not** produce this
warning and the output should look like this:
```
commit ok
pull origin
ok
pull origin
ok
push origin
```
[annex-upgrade]: https://git-annex.branchable.com/upgrades/
[deprecate-annex-issue]: https://gitlab.com/gitlab-org/gitlab/issues/1648
[git-annex]: https://git-annex.branchable.com/ "git-annex website"
[gitlab shell]: https://gitlab.com/gitlab-org/gitlab-shell "GitLab Shell repository"
[guide]: lfs/migrate_from_git_annex_to_git_lfs.html
[issue]: https://git-annex.branchable.com/forum/Error_from_git-annex-shell_on_creation_of_gcrypt_special_remote/ "git-annex issue"
[reconfigure GitLab]: restart_gitlab.md#omnibus-gitlab-reconfigure
[restart GitLab]: restart_gitlab.md#installations-from-source
doc/administration/lfs/lfs_administration.md 0 → 100644
View file @ b734365b
# GitLab Git LFS Administration
Documentation on how to use Git LFS are under [Managing large binary files with Git LFS doc](manage_large_binaries_with_git_lfs.md).
## Requirements
- Git LFS is supported in GitLab starting with version 8.2.
- Support for object storage, such as AWS S3, was introduced in 10.0.
- Users need to install [Git LFS client](https://git-lfs.github.com) version 1.0.1 and up.
## Configuration
Git LFS objects can be large in size. By default, they are stored on the server
GitLab is installed on.
There are various configuration options to help GitLab server administrators:
- Enabling/disabling Git LFS support
- Changing the location of LFS object storage
- Setting up object storage supported by [Fog](http://fog.io/about/provider_documentation.html)
### Configuration for Omnibus installations
In `/etc/gitlab/gitlab.rb`:
```ruby
# Change to true to enable lfs - enabled by default if not defined
gitlab_rails['lfs_enabled'] = false
# Optionally, change the storage path location. Defaults to
# `#{gitlab_rails['shared_path']}/lfs-objects`. Which evaluates to
# `/var/opt/gitlab/gitlab-rails/shared/lfs-objects` by default.
gitlab_rails['lfs_storage_path'] = "/mnt/storage/lfs-objects"
```
### Configuration for installations from source
In `config/gitlab.yml`:
```yaml
# Change to true to enable lfs
lfs:
enabled: false
storage_path: /mnt/storage/lfs-objects
```
## Storing LFS objects in remote object storage
> [Introduced][ee-2760] in [GitLab Premium][eep] 10.0. Brought to GitLab Core in 10.7.
It is possible to store LFS objects in remote object storage which allows you
to offload local hard disk R/W operations, and free up disk space significantly.
GitLab is tightly integrated with `Fog`, so you can refer to its [documentation](http://fog.io/about/provider_documentation.html)
to check which storage services can be integrated with GitLab.
You can also use external object storage in a private local network. For example,
[MinIO](https://min.io/) is a standalone object storage service, is easy to set up, and works well with GitLab instances.
GitLab provides two different options for the uploading mechanism: "Direct upload" and "Background upload".
**Option 1. Direct upload**
1. User pushes an lfs file to the GitLab instance
1. GitLab-workhorse uploads the file directly to the external object storage
1. GitLab-workhorse notifies GitLab-rails that the upload process is complete
**Option 2. Background upload**
1. User pushes an lfs file to the GitLab instance
1. GitLab-rails stores the file in the local file storage
1. GitLab-rails then uploads the file to the external object storage asynchronously
The following general settings are supported.
| Setting | Description | Default |
|---------|-------------|---------|
| `enabled` | Enable/disable object storage | `false` |
| `remote_directory` | The bucket name where LFS objects will be stored| |
| `direct_upload` | Set to true to enable direct upload of LFS without the need of local shared storage. Option may be removed once we decide to support only single storage for all files. | `false` |
| `background_upload` | Set to false to disable automatic upload. Option may be removed once upload is direct to S3 | `true` |
| `proxy_download` | Set to true to enable proxying all files served. Option allows to reduce egress traffic as this allows clients to download directly from remote storage instead of proxying all data | `false` |
| `connection` | Various connection options described below | |
The `connection` settings match those provided by [Fog](https://github.com/fog).
Here is a configuration example with S3.
| Setting | Description | example |
|---------|-------------|---------|
| `provider` | The provider name | AWS |
| `aws_access_key_id` | AWS credentials, or compatible | `ABC123DEF456` |
| `aws_secret_access_key` | AWS credentials, or compatible | `ABC123DEF456ABC123DEF456ABC123DEF456` |
| `aws_signature_version` | AWS signature version to use. 2 or 4 are valid options. Digital Ocean Spaces and other providers may need 2. | 4 |
| `enable_signature_v4_streaming` | Set to true to enable HTTP chunked transfers with [AWS v4 signatures](https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-streaming.html). Oracle Cloud S3 needs this to be false | true |
| `region` | AWS region | us-east-1 |
| `host` | S3 compatible host for when not using AWS, e.g. `localhost` or `storage.example.com` | s3.amazonaws.com |
| `endpoint` | Can be used when configuring an S3 compatible service such as [MinIO](https://min.io), by entering a URL such as `http://127.0.0.1:9000` | (optional) |
| `path_style` | Set to true to use `host/bucket_name/object` style paths instead of `bucket_name.host/object`. Leave as false for AWS S3 | false |
| `use_iam_profile` | Set to true to use IAM profile instead of access keys | false
Here is a configuration example with GCS.
| Setting | Description | example |
|---------|-------------|---------|
| `provider` | The provider name | `Google` |
| `google_project` | GCP project name | `gcp-project-12345` |
| `google_client_email` | The email address of the service account | `foo@gcp-project-12345.iam.gserviceaccount.com` |
| `google_json_key_location` | The json key path | `/path/to/gcp-project-12345-abcde.json` |
NOTE: **Note:**
The service account must have permission to access the bucket.
[See more](https://cloud.google.com/storage/docs/authentication)
Here is a configuration example with Rackspace Cloud Files.
| Setting | Description | example |
|---------|-------------|---------|
| `provider` | The provider name | `Rackspace` |
| `rackspace_username` | The username of the Rackspace account with access to the container | `joe.smith` |
| `rackspace_api_key` | The API key of the Rackspace account with access to the container | `ABC123DEF456ABC123DEF456ABC123DE` |
| `rackspace_region` | The Rackspace storage region to use, a three letter code from the [list of service access endpoints](https://developer.rackspace.com/docs/cloud-files/v1/general-api-info/service-access/) | `iad` |
| `rackspace_temp_url_key` | The private key you have set in the Rackspace API for temporary URLs. Read more [here](https://developer.rackspace.com/docs/cloud-files/v1/use-cases/public-access-to-your-cloud-files-account/#tempurl) | `ABC123DEF456ABC123DEF456ABC123DE` |
NOTE: **Note:**
Regardless of whether the container has public access enabled or disabled, Fog will
use the TempURL method to grant access to LFS objects. If you see errors in logs referencing
instantiating storage with a temp-url-key, ensure that you have set they key properly
on the Rackspace API and in `gitlab.rb`. You can verify the value of the key Rackspace
has set by sending a GET request with token header to the service access endpoint URL
and comparing the output of the returned headers.
### Manual uploading to an object storage
There are two ways to manually do the same thing as automatic uploading (described above).
**Option 1: rake task**
```sh
rake gitlab:lfs:migrate
```
**Option 2: rails console**
```sh
$ sudo gitlab-rails console # Login to rails console
> # Upload LFS files manually
> LfsObject.where(file_store: [nil, 1]).find_each do |lfs_object|
> lfs_object.file.migrate!(ObjectStorage::Store::REMOTE) if lfs_object.file.file.exists?
> end
```
### S3 for Omnibus installations
On Omnibus installations, the settings are prefixed by `lfs_object_store_`:
1. Edit `/etc/gitlab/gitlab.rb` and add the following lines by replacing with
the values you want:
```ruby
gitlab_rails['lfs_object_store_enabled'] = true
gitlab_rails['lfs_object_store_remote_directory'] = "lfs-objects"
gitlab_rails['lfs_object_store_connection'] = {
'provider' => 'AWS',
'region' => 'eu-central-1',
'aws_access_key_id' => '1ABCD2EFGHI34JKLM567N',
'aws_secret_access_key' => 'abcdefhijklmnopQRSTUVwxyz0123456789ABCDE',
# The below options configure an S3 compatible host instead of AWS
'host' => 'localhost',
'endpoint' => 'http://127.0.0.1:9000',
'path_style' => true
}
```
1. Save the file and [reconfigure GitLab]s for the changes to take effect.
1. Migrate any existing local LFS objects to the object storage:
```bash
gitlab-rake gitlab:lfs:migrate
```
This will migrate existing LFS objects to object storage. New LFS objects
will be forwarded to object storage unless
`gitlab_rails['lfs_object_store_background_upload']` is set to false.
### S3 for installations from source
For source installations the settings are nested under `lfs:` and then
`object_store:`:
1. Edit `/home/git/gitlab/config/gitlab.yml` and add or amend the following
lines:
```yaml
lfs:
enabled: true
object_store:
enabled: false
remote_directory: lfs-objects # Bucket name
connection:
provider: AWS
aws_access_key_id: 1ABCD2EFGHI34JKLM567N
aws_secret_access_key: abcdefhijklmnopQRSTUVwxyz0123456789ABCDE
region: eu-central-1
# Use the following options to configure an AWS compatible host such as Minio
host: 'localhost'
endpoint: 'http://127.0.0.1:9000'
path_style: true
```
1. Save the file and [restart GitLab][] for the changes to take effect.
1. Migrate any existing local LFS objects to the object storage:
```bash
sudo -u git -H bundle exec rake gitlab:lfs:migrate RAILS_ENV=production
```
This will migrate existing LFS objects to object storage. New LFS objects
will be forwarded to object storage unless `background_upload` is set to
false.
### Migrating back to local storage
In order to migrate back to local storage:
1. Set both `direct_upload` and `background_upload` to false under the LFS object storage settings. Don't forget to restart GitLab.
1. Run `rake gitlab:lfs:migrate_to_local` on your console.
1. Disable `object_storage` for LFS objects in `gitlab.rb`. Remember to restart GitLab afterwards.
## Storage statistics
You can see the total storage used for LFS objects on groups and projects
in the administration area, as well as through the [groups](../../api/groups.md)
and [projects APIs](../../api/projects.md).
## Troubleshooting: `Google::Apis::TransmissionError: execution expired`
If LFS integration is configred with Google Cloud Storage and background uploads (`background_upload: true` and `direct_upload: false`),
Sidekiq workers may encouter this error. This is because the uploading timed out with very large files.
LFS files up to 6Gb can be uploaded without any extra steps, otherwise you need to use the following workaround.
```shell
$ sudo gitlab-rails console # Login to rails console
> # Set up timeouts. 20 minutes is enough to upload 30GB LFS files.
> # These settings are only in effect for the same session, i.e. they are not effective for sidekiq workers.
> ::Google::Apis::ClientOptions.default.open_timeout_sec = 1200
> ::Google::Apis::ClientOptions.default.read_timeout_sec = 1200
> ::Google::Apis::ClientOptions.default.send_timeout_sec = 1200
> # Upload LFS files manually. This process does not use sidekiq at all.
> LfsObject.where(file_store: [nil, 1]).find_each do |lfs_object|
> lfs_object.file.migrate!(ObjectStorage::Store::REMOTE) if lfs_object.file.file.exists?
> end
```
See more information in [!19581](https://gitlab.com/gitlab-org/gitlab-foss/merge_requests/19581)
## Known limitations
- Support for removing unreferenced LFS objects was added in 8.14 onwards.
- LFS authentications via SSH was added with GitLab 8.12.
- Only compatible with the Git LFS client versions 1.1.0 and up, or 1.0.2.
- The storage statistics currently count each LFS object multiple times for
every project linking to it.
[reconfigure gitlab]: ../restart_gitlab.md#omnibus-gitlab-reconfigure "How to reconfigure Omnibus GitLab"
[restart gitlab]: ../restart_gitlab.md#installations-from-source "How to restart GitLab"
[eep]: https://about.gitlab.com/pricing/ "GitLab Premium"
[ee-2760]: https://gitlab.com/gitlab-org/gitlab/merge_requests/2760
doc/administration/lfs/manage_large_binaries_with_git_lfs.md 0 → 100644
View file @ b734365b
# Git LFS
Managing large files such as audio, video and graphics files has always been one
of the shortcomings of Git. The general recommendation is to not have Git repositories
larger than 1GB to preserve performance.
![Git LFS tracking status](img/lfs-icon.png)
An LFS icon is shown on files tracked by Git LFS to denote if a file is stored
as a blob or as an LFS pointer.
## How it works
Git LFS client talks with the GitLab server over HTTPS. It uses HTTP Basic Authentication
to authorize client requests. Once the request is authorized, Git LFS client receives
instructions from where to fetch or where to push the large file.
## GitLab server configuration
Documentation for GitLab instance administrators is under [LFS administration doc](lfs_administration.md).
## Requirements
- Git LFS is supported in GitLab starting with version 8.2
- Git LFS must be enabled under project settings
- [Git LFS client](https://git-lfs.github.com) version 1.0.1 and up
## Known limitations
- Git LFS v1 original API is not supported since it was deprecated early in LFS
development
- When SSH is set as a remote, Git LFS objects still go through HTTPS
- Any Git LFS request will ask for HTTPS credentials to be provided so a good Git
credentials store is recommended
- Git LFS always assumes HTTPS so if you have GitLab server on HTTP you will have
to add the URL to Git config manually (see [troubleshooting](#troubleshooting))
NOTE: **Note:**
With 8.12 GitLab added LFS support to SSH. The Git LFS communication
still goes over HTTP, but now the SSH client passes the correct credentials
to the Git LFS client, so no action is required by the user.
## Using Git LFS
Lets take a look at the workflow when you need to check large files into your Git
repository with Git LFS. For example, if you want to upload a very large file and
check it into your Git repository:
```bash
git clone git@gitlab.example.com:group/project.git
git lfs install # initialize the Git LFS project
git lfs track "*.iso" # select the file extensions that you want to treat as large files
```
Once a certain file extension is marked for tracking as a LFS object you can use
Git as usual without having to redo the command to track a file with the same extension:
```bash
cp ~/tmp/debian.iso ./ # copy a large file into the current directory
git add . # add the large file to the project
git commit -am "Added Debian iso" # commit the file meta data
git push origin master # sync the git repo and large file to the GitLab server
```
**Make sure** that `.gitattributes` is tracked by Git. Otherwise Git
LFS will not be working properly for people cloning the project:
```bash
git add .gitattributes
```
Cloning the repository works the same as before. Git automatically detects the
LFS-tracked files and clones them via HTTP. If you performed the `git clone`
command with a SSH URL, you have to enter your GitLab credentials for HTTP
authentication.
```bash
git clone git@gitlab.example.com:group/project.git
```
If you already cloned the repository and you want to get the latest LFS object
that are on the remote repository, eg. for a branch from origin:
```bash
git lfs fetch origin master
```
### Migrate an existing repo to Git LFS
Read the documentation on how to [migrate an existing Git repo with Git LFS](../../topics/git/migrate_to_git_lfs/index.md).
## File Locking
> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/issues/35856) in GitLab 10.5.
The first thing to do before using File Locking is to tell Git LFS which
kind of files are lockable. The following command will store PNG files
in LFS and flag them as lockable:
```bash
git lfs track "*.png" --lockable
```
After executing the above command a file named `.gitattributes` will be
created or updated with the following content:
```bash
*.png filter=lfs diff=lfs merge=lfs -text lockable
```
You can also register a file type as lockable without using LFS
(In order to be able to lock/unlock a file you need a remote server that implements the LFS File Locking API),
in order to do that you can edit the `.gitattributes` file manually:
```bash
*.pdf lockable
```
After a file type has been registered as lockable, Git LFS will make
them readonly on the file system automatically. This means you will
need to lock the file before editing it.
### Managing Locked Files
Once you're ready to edit your file you need to lock it first:
```bash
git lfs lock images/banner.png
Locked images/banner.png
```
This will register the file as locked in your name on the server:
```bash
git lfs locks
images/banner.png joe ID:123
```
Once you have pushed your changes, you can unlock the file so others can
also edit it:
```bash
git lfs unlock images/banner.png
```
You can also unlock by id:
```bash
git lfs unlock --id=123
```
If for some reason you need to unlock a file that was not locked by you,
you can use the `--force` flag as long as you have a `maintainer` access on
the project:
```bash
git lfs unlock --id=123 --force
```
## Troubleshooting
### error: Repository or object not found
There are a couple of reasons why this error can occur:
- You don't have permissions to access certain LFS object
Check if you have permissions to push to the project or fetch from the project.
- Project is not allowed to access the LFS object
LFS object you are trying to push to the project or fetch from the project is not
available to the project anymore. Probably the object was removed from the server.
- Local Git repository is using deprecated LFS API
### Invalid status for `<url>` : 501
Git LFS will log the failures into a log file.
To view this log file, while in project directory:
```bash
git lfs logs last
```
If the status `error 501` is shown, it is because:
- Git LFS is not enabled in project settings. Check your project settings and
enable Git LFS.
- Git LFS support is not enabled on the GitLab server. Check with your GitLab
administrator why Git LFS is not enabled on the server. See
[LFS administration documentation](lfs_administration.md) for instructions
on how to enable LFS support.
- Git LFS client version is not supported by GitLab server. Check your Git LFS
version with `git lfs version`. Check the Git config of the project for traces
of deprecated API with `git lfs -l`. If `batch = false` is set in the config,
remove the line and try to update your Git LFS client. Only version 1.0.1 and
newer are supported.
### getsockopt: connection refused
If you push a LFS object to a project and you receive an error similar to:
`Post <URL>/info/lfs/objects/batch: dial tcp IP: getsockopt: connection refused`,
the LFS client is trying to reach GitLab through HTTPS. However, your GitLab
instance is being served on HTTP.
This behaviour is caused by Git LFS using HTTPS connections by default when a
`lfsurl` is not set in the Git config.
To prevent this from happening, set the lfs url in project Git config:
```bash
git config --add lfs.url "http://gitlab.example.com/group/project.git/info/lfs"
```
### Credentials are always required when pushing an object
NOTE: **Note:**
With 8.12 GitLab added LFS support to SSH. The Git LFS communication
still goes over HTTP, but now the SSH client passes the correct credentials
to the Git LFS client, so no action is required by the user.
Given that Git LFS uses HTTP Basic Authentication to authenticate the user pushing
the LFS object on every push for every object, user HTTPS credentials are required.
By default, Git has support for remembering the credentials for each repository
you use. This is described in [Git credentials man pages](https://git-scm.com/docs/gitcredentials).
For example, you can tell Git to remember the password for a period of time in
which you expect to push the objects:
```bash
git config --global credential.helper 'cache --timeout=3600'
```
This will remember the credentials for an hour after which Git operations will
require re-authentication.
If you are using OS X you can use `osxkeychain` to store and encrypt your credentials.
For Windows, you can use `wincred` or Microsoft's [Git Credential Manager for Windows](https://github.com/Microsoft/Git-Credential-Manager-for-Windows/releases).
More details about various methods of storing the user credentials can be found
on [Git Credential Storage documentation](https://git-scm.com/book/en/v2/Git-Tools-Credential-Storage).
### LFS objects are missing on push
GitLab checks files to detect LFS pointers on push. If LFS pointers are detected, GitLab tries to verify that those files already exist in LFS on GitLab.
Verify that LFS in installed locally and consider a manual push with `git lfs push --all`.
If you are storing LFS files outside of GitLab you can disable LFS on the project by setting `lfs_enabled: false` with the [projects API](../../api/projects.md#edit-project).
### Hosting LFS objects externally
It is possible to host LFS objects externally by setting a custom LFS url with `git config -f .lfsconfig lfs.url https://example.com/<project>.git/info/lfs`.
You might choose to do this if you are using an appliance like a Sonatype Nexus to store LFS data. If you choose to use an external LFS store,
GitLab will not be able to verify LFS objects which means that pushes will fail if you have GitLab LFS support enabled.
To stop push failure, LFS support can be disabled in the [Project settings](../../user/project/settings/index.md). This means you will lose GitLab LFS value-adds (Verifying LFS objects, UI integration for LFS).
doc/administration/lfs/migrate_from_git_annex_to_git_lfs.md 0 → 100644
View file @ b734365b
# Migration guide from Git Annex to Git LFS
>**Note:**
Git Annex support [has been removed][issue-remove-annex] in GitLab Enterprise
Edition 9.0 (2017/03/22).
Both [Git Annex][] and [Git LFS][] are tools to manage large files in Git.
## History
Git Annex [was introduced in GitLab Enterprise Edition 7.8][post-3], at a time
where Git LFS didn't yet exist. A few months later, GitLab brought support for
Git LFS in [GitLab 8.2][post-2] and is available for both Community and
Enterprise editions.
## Differences between Git Annex and Git LFS
Some items below are general differences between the two protocols and some are
ones that GitLab developed.
- Git Annex works only through SSH, whereas Git LFS works both with SSH and HTTPS
(SSH support was added in GitLab 8.12).
- Annex files are stored in a sub-directory of the normal repositories, whereas
LFS files are stored outside of the repositories in a place you can define.
- Git Annex requires a more complex setup, but has much more options than Git
LFS. You can compare the commands each one offers by running `man git-annex`
and `man git-lfs`.
- Annex files cannot be browsed directly in GitLab's interface, whereas LFS
files can.
## Migration steps
>**Note:**
Since Git Annex files are stored in a sub-directory of the normal repositories
(`.git/annex/objects`) and LFS files are stored outside of the repositories,
they are not compatible as they are using a different scheme. Therefore, the
migration has to be done manually per repository.
There are basically two steps you need to take in order to migrate from Git
Annex to Git LFS.
### TL; DR
If you know what you are doing and want to skip the reading, this is what you
need to do (we assume you have [git-annex enabled](../git_annex.md#using-gitlab-git-annex) in your
repository and that you have made backups in case something goes wrong).
Fire up a terminal, navigate to your Git repository and:
1. Disable `git-annex`:
```bash
git annex sync --content
git annex direct
git annex uninit
git annex indirect
```
1. Enable `git-lfs`:
```
git lfs install
git lfs track <files>
git add .
git commit -m "commit message"
git push
```
### Disabling Git Annex in your repo
Before changing anything, make sure you have a backup of your repository first.
There are a couple of ways to do that, but you can simply clone it to another
local path and maybe push it to GitLab if you want a remote backup as well.
Here you'll find a guide on
[how to back up a **git-annex** repository to an external hard drive][bkp-ext-drive].
Since Annex files are stored as objects with symlinks and cannot be directly
modified, we need to first remove those symlinks.
NOTE: **Note:**
Make sure the you read about the [`direct` mode][annex-direct] as it contains
useful information that may fit in your use case. Note that `annex direct` is
deprecated in Git Annex version 6, so you may need to upgrade your repository
if the server also has Git Annex 6 installed. Read more in the
[Git Annex troubleshooting tips](../git_annex.md#troubleshooting-tips) section.
1. Backup your repository
```bash
cd repository
git annex sync --content
cd ..
git clone repository repository-backup
cd repository-backup
git annex get
cd ..
```
1. Use `annex direct`:
```bash
cd repository
git annex direct
```
The output should be similar to this:
```bash
commit
On branch master
Your branch is up-to-date with 'origin/master'.
nothing to commit, working tree clean
ok
direct debian.iso ok
direct ok
```
1. Disable Git Annex with [`annex uninit`][uninit]:
```bash
git annex uninit
```
The output should be similar to this:
```bash
unannex debian.iso ok
Deleted branch git-annex (was 2534d2c).
```
This will `unannex` every file in the repository, leaving the original files.
1. Switch back to `indirect` mode:
```bash
git annex indirect
```
The output should be similar to this:
```bash
(merging origin/git-annex into git-annex...)
(recording state in git...)
commit (recording state in git...)
ok
(recording state in git...)
[master fac3194] commit before switching to indirect mode
1 file changed, 1 deletion(-)
delete mode 120000 alpine-virt-3.4.4-x86_64.iso
ok
indirect ok
ok
```
---
At this point, you have two options. Either add, commit and push the files
directly back to GitLab or switch to Git LFS. We will tackle the LFS switch in
the next section.
### Enabling Git LFS in your repo
Git LFS is enabled by default on all GitLab products (GitLab CE, GitLab EE,
GitLab.com), therefore, you don't need to do anything server-side.
1. First, make sure you have `git-lfs` installed locally:
```bash
git lfs help
```
If the terminal doesn't prompt you with a full response on `git-lfs` commands,
[install the Git LFS client][install-lfs] first.
1. Inside the repo, run the following command to initiate LFS:
```bash
git lfs install
```
1. Enable `git-lfs` for the group of files you want to track. You
can track specific files, all files containing the same extension, or an
entire directory:
```bash
git lfs track images/01.png # per file
git lfs track **/*.png # per extension
git lfs track images/ # per directory
```
Once you do that, run `git status` and you'll see `.gitattributes` added
to your repo. It collects all file patterns that you chose to track via
`git-lfs`.
1. Add the files, commit and push them to GitLab:
```bash
git add .
git commit -m "commit message"
git push
```
If your remote is set up with HTTP, you will be asked to enter your login
credentials. If you have [2FA enabled](../../user/profile/account/two_factor_authentication.md), make sure to use a
[personal access token](../../user/profile/account/two_factor_authentication.md#personal-access-tokens)
instead of your password.
## Removing the Git Annex branches
After the migration finishes successfully, you can remove all `git-annex`
related branches from your repository.
On GitLab, navigate to your project's **Repository ➔ Branches** and delete all
branches created by Git Annex: `git-annex`, and all under `synced/`.
![repository branches](img/git-annex-branches.png)
You can also do this on the command line with:
```bash
git branch -d synced/master
git branch -d synced/git-annex
git push origin :synced/master
git push origin :synced/git-annex
git push origin :git-annex
git remote prune origin
```
If there are still some Annex objects inside your repository (`.git/annex/`)
or references inside `.git/config`, run `annex uninit` again:
```bash
git annex uninit
```
## Further Reading
- (Blog Post) [Getting Started with Git FLS][post-1]
- (Blog Post) [Announcing LFS Support in GitLab][post-2]
- (Blog Post) [GitLab Annex Solves the Problem of Versioning Large Binaries with Git][post-3]
- (GitLab Docs) [Git Annex](../git_annex.md)
- (GitLab Docs) [Git LFS](manage_large_binaries_with_git_lfs.md)
[annex-direct]: https://git-annex.branchable.com/direct_mode/
[bkp-ext-drive]: https://www.thomas-krenn.com/en/wiki/Git-annex_Repository_on_an_External_Hard_Drive
[Git Annex]: http://git-annex.branchable.com/
[Git LFS]: https://git-lfs.github.com/
[install-lfs]: https://git-lfs.github.com/
[issue-remove-annex]: https://gitlab.com/gitlab-org/gitlab/issues/1648
[lfs-track]: https://about.gitlab.com/blog/2017/01/30/getting-started-with-git-lfs-tutorial/#tracking-files-with-lfs
[post-1]: https://about.gitlab.com/blog/2017/01/30/getting-started-with-git-lfs-tutorial/
[post-2]: https://about.gitlab.com/blog/2015/11/23/announcing-git-lfs-support-in-gitlab/
[post-3]: https://about.gitlab.com/blog/2015/02/17/gitlab-annex-solves-the-problem-of-versioning-large-binaries-with-git/
[uninit]: https://git-annex.branchable.com/git-annex-uninit/
doc/development/contributing/merge_request_workflow.md
View file @ b734365b
...@@ -36,7 +36,7 @@ include a regression test are merged quickly, while new features without proper ...@@ -36,7 +36,7 @@ include a regression test are merged quickly, while new features without proper
tests might be slower to receive feedback. The workflow to make a merge tests might be slower to receive feedback. The workflow to make a merge
request is as follows: request is as follows:
1. [Fork](../../workflow/forking_workflow.md#creating-a-fork) the project into 1. [Fork](../../user/project/forking_workflow.md) the project into
your personal namespace (or group) on GitLab.com. your personal namespace (or group) on GitLab.com.
1. Create a feature branch in your fork (don't work off `master`). 1. Create a feature branch in your fork (don't work off `master`).
1. Write [tests](../rake_tasks.md#run-tests) and code. 1. Write [tests](../rake_tasks.md#run-tests) and code.
......
doc/development/git_object_deduplication.md
View file @ b734365b
# How Git object deduplication works in GitLab # How Git object deduplication works in GitLab
When a GitLab user [forks a project](../workflow/forking_workflow.md), When a GitLab user [forks a project](../user/project/forking_workflow.md),
GitLab creates a new Project with an associated Git repository that is a GitLab creates a new Project with an associated Git repository that is a
copy of the original project at the time of the fork. If a large project copy of the original project at the time of the fork. If a large project
gets forked often, this can lead to a quick increase in Git repository gets forked often, this can lead to a quick increase in Git repository
......
doc/gitlab-basics/fork-project.md
View file @ b734365b
...@@ -8,4 +8,4 @@ A fork is a copy of an original repository that you put in another namespace ...@@ -8,4 +8,4 @@ A fork is a copy of an original repository that you put in another namespace
where you can experiment and apply changes that you can later decide whether or where you can experiment and apply changes that you can later decide whether or
not to share, without affecting the original project. not to share, without affecting the original project.
It takes just a few steps to [fork a project in GitLab](../workflow/forking_workflow.md#creating-a-fork). It takes just a few steps to [fork a project in GitLab](../user/project/forking_workflow.md#creating-a-fork).
doc/intro/README.md
View file @ b734365b
...@@ -24,7 +24,7 @@ Create issues, labels, milestones, cast your vote, and review issues. ...@@ -24,7 +24,7 @@ Create issues, labels, milestones, cast your vote, and review issues.
Create merge requests and review code. Create merge requests and review code.
- [Fork a project and contribute to it](../workflow/forking_workflow.md) - [Fork a project and contribute to it](../user/project/forking_workflow.md)
- [Create a new merge request](../gitlab-basics/add-merge-request.md) - [Create a new merge request](../gitlab-basics/add-merge-request.md)
- [Automatically close issues from merge requests](../user/project/issues/managing_issues.md#closing-issues-automatically) - [Automatically close issues from merge requests](../user/project/issues/managing_issues.md#closing-issues-automatically)
- [Automatically merge when pipeline succeeds](../user/project/merge_requests/merge_when_pipeline_succeeds.md) - [Automatically merge when pipeline succeeds](../user/project/merge_requests/merge_when_pipeline_succeeds.md)
......
doc/topics/git/migrate_to_git_lfs/index.md
View file @ b734365b
...@@ -163,9 +163,9 @@ but commented out to help encourage others to add to it in the future. --> ...@@ -163,9 +163,9 @@ but commented out to help encourage others to add to it in the future. -->
## References ## References
- [Getting Started with Git LFS](https://about.gitlab.com/blog/2017/01/30/getting-started-with-git-lfs-tutorial/) - [Getting Started with Git LFS](https://about.gitlab.com/blog/2017/01/30/getting-started-with-git-lfs-tutorial/)
- [Migrate from Git Annex to Git LFS](../../../workflow/lfs/migrate_from_git_annex_to_git_lfs.md) - [Migrate from Git Annex to Git LFS](../../../administration/lfs/migrate_from_git_annex_to_git_lfs.md)
- [GitLab's Git LFS user documentation](../../../workflow/lfs/manage_large_binaries_with_git_lfs.md) - [GitLab's Git LFS user documentation](../../../administration/lfs/manage_large_binaries_with_git_lfs.md)
- [GitLab's Git LFS administrator documentation](../../../workflow/lfs/lfs_administration.md) - [GitLab's Git LFS administrator documentation](../../../administration/lfs/lfs_administration.md)
- Alternative method to [migrate an existing repo to Git LFS](https://github.com/git-lfs/git-lfs/wiki/Tutorial#migrating-existing-repository-data-to-lfs) - Alternative method to [migrate an existing repo to Git LFS](https://github.com/git-lfs/git-lfs/wiki/Tutorial#migrating-existing-repository-data-to-lfs)
<!-- <!--
......
doc/user/project/forking_workflow.md 0 → 100644
View file @ b734365b
# Project forking workflow
Forking a project to your own namespace is useful if you have no write
access to the project you want to contribute to. If you do have write
access or can request it, we recommend working together in the same
repository since it is simpler. See our [GitLab Flow](../../workflow/gitlab_flow.md)
document more information about using branches to work together.
## Creating a fork
Forking a project is in most cases a two-step process.
1. Click on the fork button located located in between the star and clone buttons on the project's home page.
![Fork button](img/forking_workflow_fork_button.png)
1. Once you do that, you'll be presented with a screen where you can choose
the namespace to fork to. Only namespaces (groups and your own
namespace) where you have write access to, will be shown. Click on the
namespace to create your fork there.
![Choose namespace](img/forking_workflow_choose_namespace.png)
**Note:**
If the namespace you chose to fork the project to has another project with
the same path name, you will be presented with a warning that the forking
could not be completed. Try to resolve the error before repeating the forking
process.
![Path taken error](img/forking_workflow_path_taken_error.png)
After the forking is done, you can start working on the newly created
repository. There, you will have full [Owner](../permissions.md)
access, so you can set it up as you please.
## Merging upstream
Once you are ready to send your code back to the main project, you need
to create a merge request. Choose your forked project's main branch as
the source and the original project's main branch as the destination and
create the [merge request](merge_requests/index.md).
![Selecting branches](img/forking_workflow_branch_select.png)
You can then assign the merge request to someone to have them review
your changes. Upon pressing the 'Submit Merge Request' button, your
changes will be added to the repository and branch you're merging into.
![New merge request](img/forking_workflow_merge_request.png)
[gitlab flow]: https://about.gitlab.com/blog/2014/09/29/gitlab-flow/ "GitLab Flow blog post"
doc/user/project/repository/file_finder.md 0 → 100644
View file @ b734365b
# File finder
> [Introduced][gh-9889] in GitLab 8.4.
The file finder feature allows you to quickly shortcut your way when you are
searching for a file in a repository using the GitLab UI.
You can find the **Find File** button when in the **Files** section of a
project.
![Find file button](img/file_finder_find_button.png)
For those who prefer to keep their fingers on the keyboard, there is a
[shortcut button](../../../workflow/shortcuts.md) as well, which you can invoke from _anywhere_
in a project.
Press `t` to launch the File search function when in **Issues**,
**Merge requests**, **Milestones**, even the project's settings.
Start typing what you are searching for and watch the magic happen. With the
up/down arrows, you go up and down the results, with `Esc` you close the search
and go back to **Files**.
## How it works
The File finder feature is powered by the [Fuzzy filter](https://github.com/jeancroy/fuzz-aldrin-plus) library.
It implements a fuzzy search with highlight, and tries to provide intuitive
results by recognizing patterns that people use while searching.
For example, consider the [GitLab CE repository][ce] and that we want to open
the `app/controllers/admin/deploy_keys_controller.rb` file.
Using fuzzy search, we start by typing letters that get us closer to the file.
**Protip:** To narrow down your search, include `/` in your search terms.
![Find file button](img/file_finder_find_file.png)
[gh-9889]: https://github.com/gitlabhq/gitlabhq/pull/9889 "File finder pull request"
[ce]: https://gitlab.com/gitlab-org/gitlab-foss/tree/master "GitLab CE repository"
doc/user/project/repository/index.md
View file @ b734365b
...@@ -55,7 +55,7 @@ To get started with the command line, please read through the ...@@ -55,7 +55,7 @@ To get started with the command line, please read through the
### Find files ### Find files
Use GitLab's [file finder](../../../workflow/file_finder.md) to search for files in a repository. Use GitLab's [file finder](file_finder.md) to search for files in a repository.
### Supported markup languages and extensions ### Supported markup languages and extensions
......
doc/workflow/README.md
View file @ b734365b
...@@ -18,13 +18,13 @@ comments: false ...@@ -18,13 +18,13 @@ comments: false
- [Due date for issues](../user/project/issues/due_dates.md) - [Due date for issues](../user/project/issues/due_dates.md)
- [Issue Board](../user/project/issue_board.md) - [Issue Board](../user/project/issue_board.md)
- [Keyboard shortcuts](shortcuts.md) - [Keyboard shortcuts](shortcuts.md)
- [File finder](file_finder.md) - [File finder](../user/project/repository/file_finder.md)
- [File lock](../user/project/file_lock.md) **(PREMIUM)** - [File lock](../user/project/file_lock.md) **(PREMIUM)**
- [Labels](../user/project/labels.md) - [Labels](../user/project/labels.md)
- [Issue weight](issue_weight.md) **(STARTER)** - [Issue weight](issue_weight.md) **(STARTER)**
- [Notification emails](notifications.md) - [Notification emails](notifications.md)
- [Projects](../user/project/index.md) - [Projects](../user/project/index.md)
- [Project forking workflow](forking_workflow.md) - [Project forking workflow](../user/project/forking_workflow.md)
- [Project users](../user/project/members/index.md) - [Project users](../user/project/members/index.md)
- [Protected branches](../user/project/protected_branches.md) - [Protected branches](../user/project/protected_branches.md)
- [Protected tags](../user/project/protected_tags.md) - [Protected tags](../user/project/protected_tags.md)
...@@ -47,7 +47,7 @@ comments: false ...@@ -47,7 +47,7 @@ comments: false
- [Merge request approvals](../user/project/merge_requests/merge_request_approvals.md) **(STARTER)** - [Merge request approvals](../user/project/merge_requests/merge_request_approvals.md) **(STARTER)**
- [Repository mirroring](repository_mirroring.md) **(STARTER)** - [Repository mirroring](repository_mirroring.md) **(STARTER)**
- [Service Desk](../user/project/service_desk.md) **(PREMIUM)** - [Service Desk](../user/project/service_desk.md) **(PREMIUM)**
- [Manage large binaries with Git LFS](lfs/manage_large_binaries_with_git_lfs.md) - [Manage large binaries with Git LFS](../administration/lfs/manage_large_binaries_with_git_lfs.md)
- [Importing from SVN, GitHub, Bitbucket, etc](importing/README.md) - [Importing from SVN, GitHub, Bitbucket, etc](importing/README.md)
- [Todos](todos.md) - [Todos](todos.md)
- [Snippets](../user/snippets.md) - [Snippets](../user/snippets.md)
......
doc/workflow/file_finder.md
View file @ b734365b
# File finder ---
redirect_to: '../user/project/repository/file_finder.md'
---
> [Introduced][gh-9889] in GitLab 8.4. This document was moved to [another location](../user/project/repository/file_finder.md).
The file finder feature allows you to quickly shortcut your way when you are
searching for a file in a repository using the GitLab UI.
You can find the **Find File** button when in the **Files** section of a
project.
![Find file button](img/file_finder_find_button.png)
For those who prefer to keep their fingers on the keyboard, there is a
[shortcut button](shortcuts.md) as well, which you can invoke from _anywhere_
in a project.
Press `t` to launch the File search function when in **Issues**,
**Merge requests**, **Milestones**, even the project's settings.
Start typing what you are searching for and watch the magic happen. With the
up/down arrows, you go up and down the results, with `Esc` you close the search
and go back to **Files**.
## How it works
The File finder feature is powered by the [Fuzzy filter](https://github.com/jeancroy/fuzz-aldrin-plus) library.
It implements a fuzzy search with highlight, and tries to provide intuitive
results by recognizing patterns that people use while searching.
For example, consider the [GitLab CE repository][ce] and that we want to open
the `app/controllers/admin/deploy_keys_controller.rb` file.
Using fuzzy search, we start by typing letters that get us closer to the file.
**Protip:** To narrow down your search, include `/` in your search terms.
![Find file button](img/file_finder_find_file.png)
[gh-9889]: https://github.com/gitlabhq/gitlabhq/pull/9889 "File finder pull request"
[ce]: https://gitlab.com/gitlab-org/gitlab-foss/tree/master "GitLab CE repository"
doc/workflow/forking_workflow.md
View file @ b734365b
# Project forking workflow ---
redirect_to: '../user/project/forking_workflow.md'
---
Forking a project to your own namespace is useful if you have no write This document was moved to [another location](../user/project/forking_workflow.md).
access to the project you want to contribute to. If you do have write
access or can request it, we recommend working together in the same
repository since it is simpler. See our [GitLab Flow](gitlab_flow.md)
document more information about using branches to work together.
## Creating a fork
Forking a project is in most cases a two-step process.
1. Click on the fork button located located in between the star and clone buttons on the project's home page.
![Fork button](img/forking_workflow_fork_button.png)
1. Once you do that, you'll be presented with a screen where you can choose
the namespace to fork to. Only namespaces (groups and your own
namespace) where you have write access to, will be shown. Click on the
namespace to create your fork there.
![Choose namespace](img/forking_workflow_choose_namespace.png)
**Note:**
If the namespace you chose to fork the project to has another project with
the same path name, you will be presented with a warning that the forking
could not be completed. Try to resolve the error before repeating the forking
process.
![Path taken error](img/forking_workflow_path_taken_error.png)
After the forking is done, you can start working on the newly created
repository. There, you will have full [Owner](../user/permissions.md)
access, so you can set it up as you please.
## Merging upstream
Once you are ready to send your code back to the main project, you need
to create a merge request. Choose your forked project's main branch as
the source and the original project's main branch as the destination and
create the [merge request](merge_requests.md).
![Selecting branches](forking/branch_select.png)
You can then assign the merge request to someone to have them review
your changes. Upon pressing the 'Submit Merge Request' button, your
changes will be added to the repository and branch you're merging into.
![New merge request](forking/merge_request.png)
[gitlab flow]: https://about.gitlab.com/blog/2014/09/29/gitlab-flow/ "GitLab Flow blog post"
doc/workflow/git_annex.md
View file @ b734365b
# Git annex ---
redirect_to: '../administration/git_annex.md'
> **Warning:** GitLab has [completely
removed][deprecate-annex-issue] in GitLab 9.0 (2017/03/22).
Read through the [migration guide from git-annex to Git LFS][guide].
The biggest limitation of Git, compared to some older centralized version
control systems, has been the maximum size of the repositories.
The general recommendation is to not have Git repositories larger than 1GB to
preserve performance. Although GitLab has no limit (some repositories in GitLab
are over 50GB!), we subscribe to the advice to keep repositories as small as
you can.
Not being able to version control large binaries is a big problem for many
larger organizations.
Videos, photos, audio, compiled binaries and many other types of files are too
large. As a workaround, people keep artwork-in-progress in a Dropbox folder and
only check in the final result. This results in using outdated files, not
having a complete history and increases the risk of losing work.
This problem is solved in GitLab Enterprise Edition by integrating the
[git-annex] application.
`git-annex` allows managing large binaries with Git without checking the
contents into Git.
You check-in only a symlink that contains the SHA-1 of the large binary. If you
need the large binary, you can sync it from the GitLab server over `rsync`, a
very fast file copying tool.
## GitLab git-annex Configuration
`git-annex` is disabled by default in GitLab. Below you will find the
configuration options required to enable it.
### Requirements
`git-annex` needs to be installed both on the server and the client side.
For Debian-like systems (e.g., Debian, Ubuntu) this can be achieved by running:
```
sudo apt-get update && sudo apt-get install git-annex
```
For RedHat-like systems (e.g., CentOS, RHEL) this can be achieved by running:
```
sudo yum install epel-release && sudo yum install git-annex
```
### Configuration for Omnibus packages
For Omnibus GitLab packages, only one configuration setting is needed.
The Omnibus package will internally set the correct options in all locations.
1. In `/etc/gitlab/gitlab.rb` add the following line:
```ruby
gitlab_shell['git_annex_enabled'] = true
```
1. Save the file and [reconfigure GitLab][] for the changes to take effect.
### Configuration for installations from source
There are 2 settings to enable git-annex on your GitLab server.
One is located in `config/gitlab.yml` of the GitLab repository and the other
one is located in `config.yml` of GitLab Shell.
1. In `config/gitlab.yml` add or edit the following lines:
```yaml
gitlab_shell:
git_annex_enabled: true
```
1. In `config.yml` of GitLab Shell add or edit the following lines:
```yaml
git_annex_enabled: true
```
1. Save the files and [restart GitLab][] for the changes to take effect.
## Using GitLab git-annex
> **Note:**
> Your Git remotes must be using the SSH protocol, not HTTP(S).
Here is an example workflow of uploading a very large file and then checking it
into your Git repository:
```bash
git clone git@example.com:group/project.git
git annex init 'My Laptop' # initialize the annex project and give an optional description
cp ~/tmp/debian.iso ./ # copy a large file into the current directory
git annex add debian.iso # add the large file to git annex
git commit -am "Add Debian iso" # commit the file metadata
git annex sync --content # sync the Git repo and large file to the GitLab server
```
The output should look like this:
```
commit
On branch master
Your branch is ahead of 'origin/master' by 1 commit.
(use "git push" to publish your local commits)
nothing to commit, working tree clean
ok
pull origin
remote: Counting objects: 5, done.
remote: Compressing objects: 100% (4/4), done.
remote: Total 5 (delta 2), reused 0 (delta 0)
Unpacking objects: 100% (5/5), done.
From example.com:group/project
497842b..5162f80 git-annex -> origin/git-annex
ok
(merging origin/git-annex into git-annex...)
(recording state in git...)
copy debian.iso (checking origin...) (to origin...)
SHA256E-s26214400--8092b3d482fb1b7a5cf28c43bc1425c8f2d380e86869c0686c49aa7b0f086ab2.iso
26,214,400 100% 638.88kB/s 0:00:40 (xfr#1, to-chk=0/1)
ok
pull origin
ok
(recording state in git...)
push origin
Counting objects: 15, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (13/13), done.
Writing objects: 100% (15/15), 1.64 KiB | 0 bytes/s, done.
Total 15 (delta 1), reused 0 (delta 0)
To example.com:group/project.git
* [new branch] git-annex -> synced/git-annex
* [new branch] master -> synced/master
ok
```
Your files can be found in the `master` branch, but you'll notice that there
are more branches created by the `annex sync` command.
Git Annex will also create a new directory at `.git/annex/` and will record the
tracked files in the `.git/config` file. The files you assign to be tracked
with `git-annex` will not affect the existing `.git/config` records. The files
are turned into symbolic links that point to data in `.git/annex/objects/`.
The `debian.iso` file in the example will contain the symbolic link:
```
.git/annex/objects/ZW/1k/SHA256E-s82701--6384039733b5035b559efd5a2e25a493ab6e09aabfd5162cc03f6f0ec238429d.png/SHA256E-s82701--6384039733b5035b559efd5a2e25a493ab6e09aabfd5162cc03f6f0ec238429d.iso
```
Use `git annex info` to retrieve the information about the local copy of your
repository.
--- ---
Downloading a single large file is also very simple: This document was moved to [another location](../administration/git_annex.md).
```bash
git clone git@gitlab.example.com:group/project.git
git annex sync # sync Git branches but not the large file
git annex get debian.iso # download the large file
```
To download all files:
```bash
git clone git@gitlab.example.com:group/project.git
git annex sync --content # sync Git branches and download all the large files
```
By using `git-annex` without GitLab, anyone that can access the server can also
access the files of all projects, but GitLab Annex ensures that you can only
access files of projects you have access to (developer, maintainer, or owner role).
## How it works
Internally GitLab uses [GitLab Shell] to handle SSH access and this was a great
integration point for `git-annex`.
There is a setting in GitLab Shell so you can disable GitLab Annex support
if you want to.
## Troubleshooting tips
Differences in version of `git-annex` on the GitLab server and on local machines
can cause `git-annex` to raise unpredicted warnings and errors.
Consult the [Annex upgrade page][annex-upgrade] for more information about
the differences between versions. You can find out which version is installed
on your server by navigating to <https://pkgs.org/download/git-annex> and
searching for your distribution.
Although there is no general guide for `git-annex` errors, there are a few tips
on how to go around the warnings.
### `git-annex-shell: Not a git-annex or gcrypt repository`
This warning can appear on the initial `git annex sync --content` and is caused
by differences in `git-annex-shell`. You can read more about it
[in this git-annex issue][issue].
One important thing to note is that despite the warning, the `sync` succeeds
and the files are pushed to the GitLab repository.
If you get hit by this, you can run the following command inside the repository
that the warning was raised:
```
git config remote.origin.annex-ignore false
```
Consecutive runs of `git annex sync --content` **should not** produce this
warning and the output should look like this:
```
commit ok
pull origin
ok
pull origin
ok
push origin
```
[annex-upgrade]: https://git-annex.branchable.com/upgrades/
[deprecate-annex-issue]: https://gitlab.com/gitlab-org/gitlab/issues/1648
[git-annex]: https://git-annex.branchable.com/ "git-annex website"
[gitlab shell]: https://gitlab.com/gitlab-org/gitlab-shell "GitLab Shell repository"
[guide]: lfs/migrate_from_git_annex_to_git_lfs.html
[issue]: https://git-annex.branchable.com/forum/Error_from_git-annex-shell_on_creation_of_gcrypt_special_remote/ "git-annex issue"
[reconfigure GitLab]: ../administration/restart_gitlab.md#omnibus-gitlab-reconfigure
[restart GitLab]: ../administration/restart_gitlab.md#installations-from-source
doc/workflow/git_lfs.md
View file @ b734365b
--- ---
redirect_to: 'lfs/manage_large_binaries_with_git_lfs.md' redirect_to: '../administration/lfs/manage_large_binaries_with_git_lfs.md'
--- ---
This document was moved to [another location](lfs/manage_large_binaries_with_git_lfs.md). This document was moved to [another location](../administration/lfs/manage_large_binaries_with_git_lfs.md).
doc/workflow/lfs/lfs_administration.md
View file @ b734365b
# GitLab Git LFS Administration ---
redirect_to: '../../administration/lfs/lfs_administration.md'
---
Documentation on how to use Git LFS are under [Managing large binary files with Git LFS doc](manage_large_binaries_with_git_lfs.md). This document was moved to [another location](../../administration/lfs/lfs_administration.md).
## Requirements
- Git LFS is supported in GitLab starting with version 8.2.
- Support for object storage, such as AWS S3, was introduced in 10.0.
- Users need to install [Git LFS client](https://git-lfs.github.com) version 1.0.1 and up.
## Configuration
Git LFS objects can be large in size. By default, they are stored on the server
GitLab is installed on.
There are various configuration options to help GitLab server administrators:
- Enabling/disabling Git LFS support
- Changing the location of LFS object storage
- Setting up object storage supported by [Fog](http://fog.io/about/provider_documentation.html)
### Configuration for Omnibus installations
In `/etc/gitlab/gitlab.rb`:
```ruby
# Change to true to enable lfs - enabled by default if not defined
gitlab_rails['lfs_enabled'] = false
# Optionally, change the storage path location. Defaults to
# `#{gitlab_rails['shared_path']}/lfs-objects`. Which evaluates to
# `/var/opt/gitlab/gitlab-rails/shared/lfs-objects` by default.
gitlab_rails['lfs_storage_path'] = "/mnt/storage/lfs-objects"
```
### Configuration for installations from source
In `config/gitlab.yml`:
```yaml
# Change to true to enable lfs
lfs:
enabled: false
storage_path: /mnt/storage/lfs-objects
```
## Storing LFS objects in remote object storage
> [Introduced][ee-2760] in [GitLab Premium][eep] 10.0. Brought to GitLab Core in 10.7.
It is possible to store LFS objects in remote object storage which allows you
to offload local hard disk R/W operations, and free up disk space significantly.
GitLab is tightly integrated with `Fog`, so you can refer to its [documentation](http://fog.io/about/provider_documentation.html)
to check which storage services can be integrated with GitLab.
You can also use external object storage in a private local network. For example,
[MinIO](https://min.io/) is a standalone object storage service, is easy to set up, and works well with GitLab instances.
GitLab provides two different options for the uploading mechanism: "Direct upload" and "Background upload".
**Option 1. Direct upload**
1. User pushes an lfs file to the GitLab instance
1. GitLab-workhorse uploads the file directly to the external object storage
1. GitLab-workhorse notifies GitLab-rails that the upload process is complete
**Option 2. Background upload**
1. User pushes an lfs file to the GitLab instance
1. GitLab-rails stores the file in the local file storage
1. GitLab-rails then uploads the file to the external object storage asynchronously
The following general settings are supported.
| Setting | Description | Default |
|---------|-------------|---------|
| `enabled` | Enable/disable object storage | `false` |
| `remote_directory` | The bucket name where LFS objects will be stored| |
| `direct_upload` | Set to true to enable direct upload of LFS without the need of local shared storage. Option may be removed once we decide to support only single storage for all files. | `false` |
| `background_upload` | Set to false to disable automatic upload. Option may be removed once upload is direct to S3 | `true` |
| `proxy_download` | Set to true to enable proxying all files served. Option allows to reduce egress traffic as this allows clients to download directly from remote storage instead of proxying all data | `false` |
| `connection` | Various connection options described below | |
The `connection` settings match those provided by [Fog](https://github.com/fog).
Here is a configuration example with S3.
| Setting | Description | example |
|---------|-------------|---------|
| `provider` | The provider name | AWS |
| `aws_access_key_id` | AWS credentials, or compatible | `ABC123DEF456` |
| `aws_secret_access_key` | AWS credentials, or compatible | `ABC123DEF456ABC123DEF456ABC123DEF456` |
| `aws_signature_version` | AWS signature version to use. 2 or 4 are valid options. Digital Ocean Spaces and other providers may need 2. | 4 |
| `enable_signature_v4_streaming` | Set to true to enable HTTP chunked transfers with [AWS v4 signatures](https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-streaming.html). Oracle Cloud S3 needs this to be false | true |
| `region` | AWS region | us-east-1 |
| `host` | S3 compatible host for when not using AWS, e.g. `localhost` or `storage.example.com` | s3.amazonaws.com |
| `endpoint` | Can be used when configuring an S3 compatible service such as [MinIO](https://min.io), by entering a URL such as `http://127.0.0.1:9000` | (optional) |
| `path_style` | Set to true to use `host/bucket_name/object` style paths instead of `bucket_name.host/object`. Leave as false for AWS S3 | false |
| `use_iam_profile` | Set to true to use IAM profile instead of access keys | false
Here is a configuration example with GCS.
| Setting | Description | example |
|---------|-------------|---------|
| `provider` | The provider name | `Google` |
| `google_project` | GCP project name | `gcp-project-12345` |
| `google_client_email` | The email address of the service account | `foo@gcp-project-12345.iam.gserviceaccount.com` |
| `google_json_key_location` | The json key path | `/path/to/gcp-project-12345-abcde.json` |
NOTE: **Note:**
The service account must have permission to access the bucket.
[See more](https://cloud.google.com/storage/docs/authentication)
Here is a configuration example with Rackspace Cloud Files.
| Setting | Description | example |
|---------|-------------|---------|
| `provider` | The provider name | `Rackspace` |
| `rackspace_username` | The username of the Rackspace account with access to the container | `joe.smith` |
| `rackspace_api_key` | The API key of the Rackspace account with access to the container | `ABC123DEF456ABC123DEF456ABC123DE` |
| `rackspace_region` | The Rackspace storage region to use, a three letter code from the [list of service access endpoints](https://developer.rackspace.com/docs/cloud-files/v1/general-api-info/service-access/) | `iad` |
| `rackspace_temp_url_key` | The private key you have set in the Rackspace API for temporary URLs. Read more [here](https://developer.rackspace.com/docs/cloud-files/v1/use-cases/public-access-to-your-cloud-files-account/#tempurl) | `ABC123DEF456ABC123DEF456ABC123DE` |
NOTE: **Note:**
Regardless of whether the container has public access enabled or disabled, Fog will
use the TempURL method to grant access to LFS objects. If you see errors in logs referencing
instantiating storage with a temp-url-key, ensure that you have set they key properly
on the Rackspace API and in `gitlab.rb`. You can verify the value of the key Rackspace
has set by sending a GET request with token header to the service access endpoint URL
and comparing the output of the returned headers.
### Manual uploading to an object storage
There are two ways to manually do the same thing as automatic uploading (described above).
**Option 1: rake task**
```sh
rake gitlab:lfs:migrate
```
**Option 2: rails console**
```sh
$ sudo gitlab-rails console # Login to rails console
> # Upload LFS files manually
> LfsObject.where(file_store: [nil, 1]).find_each do |lfs_object|
> lfs_object.file.migrate!(ObjectStorage::Store::REMOTE) if lfs_object.file.file.exists?
> end
```
### S3 for Omnibus installations
On Omnibus installations, the settings are prefixed by `lfs_object_store_`:
1. Edit `/etc/gitlab/gitlab.rb` and add the following lines by replacing with
the values you want:
```ruby
gitlab_rails['lfs_object_store_enabled'] = true
gitlab_rails['lfs_object_store_remote_directory'] = "lfs-objects"
gitlab_rails['lfs_object_store_connection'] = {
'provider' => 'AWS',
'region' => 'eu-central-1',
'aws_access_key_id' => '1ABCD2EFGHI34JKLM567N',
'aws_secret_access_key' => 'abcdefhijklmnopQRSTUVwxyz0123456789ABCDE',
# The below options configure an S3 compatible host instead of AWS
'host' => 'localhost',
'endpoint' => 'http://127.0.0.1:9000',
'path_style' => true
}
```
1. Save the file and [reconfigure GitLab]s for the changes to take effect.
1. Migrate any existing local LFS objects to the object storage:
```bash
gitlab-rake gitlab:lfs:migrate
```
This will migrate existing LFS objects to object storage. New LFS objects
will be forwarded to object storage unless
`gitlab_rails['lfs_object_store_background_upload']` is set to false.
### S3 for installations from source
For source installations the settings are nested under `lfs:` and then
`object_store:`:
1. Edit `/home/git/gitlab/config/gitlab.yml` and add or amend the following
lines:
```yaml
lfs:
enabled: true
object_store:
enabled: false
remote_directory: lfs-objects # Bucket name
connection:
provider: AWS
aws_access_key_id: 1ABCD2EFGHI34JKLM567N
aws_secret_access_key: abcdefhijklmnopQRSTUVwxyz0123456789ABCDE
region: eu-central-1
# Use the following options to configure an AWS compatible host such as Minio
host: 'localhost'
endpoint: 'http://127.0.0.1:9000'
path_style: true
```
1. Save the file and [restart GitLab][] for the changes to take effect.
1. Migrate any existing local LFS objects to the object storage:
```bash
sudo -u git -H bundle exec rake gitlab:lfs:migrate RAILS_ENV=production
```
This will migrate existing LFS objects to object storage. New LFS objects
will be forwarded to object storage unless `background_upload` is set to
false.
### Migrating back to local storage
In order to migrate back to local storage:
1. Set both `direct_upload` and `background_upload` to false under the LFS object storage settings. Don't forget to restart GitLab.
1. Run `rake gitlab:lfs:migrate_to_local` on your console.
1. Disable `object_storage` for LFS objects in `gitlab.rb`. Remember to restart GitLab afterwards.
## Storage statistics
You can see the total storage used for LFS objects on groups and projects
in the administration area, as well as through the [groups](../../api/groups.md)
and [projects APIs](../../api/projects.md).
## Troubleshooting: `Google::Apis::TransmissionError: execution expired`
If LFS integration is configred with Google Cloud Storage and background uploads (`background_upload: true` and `direct_upload: false`),
Sidekiq workers may encouter this error. This is because the uploading timed out with very large files.
LFS files up to 6Gb can be uploaded without any extra steps, otherwise you need to use the following workaround.
```shell
$ sudo gitlab-rails console # Login to rails console
> # Set up timeouts. 20 minutes is enough to upload 30GB LFS files.
> # These settings are only in effect for the same session, i.e. they are not effective for sidekiq workers.
> ::Google::Apis::ClientOptions.default.open_timeout_sec = 1200
> ::Google::Apis::ClientOptions.default.read_timeout_sec = 1200
> ::Google::Apis::ClientOptions.default.send_timeout_sec = 1200
> # Upload LFS files manually. This process does not use sidekiq at all.
> LfsObject.where(file_store: [nil, 1]).find_each do |lfs_object|
> lfs_object.file.migrate!(ObjectStorage::Store::REMOTE) if lfs_object.file.file.exists?
> end
```
See more information in [!19581](https://gitlab.com/gitlab-org/gitlab-foss/merge_requests/19581)
## Known limitations
- Support for removing unreferenced LFS objects was added in 8.14 onwards.
- LFS authentications via SSH was added with GitLab 8.12.
- Only compatible with the Git LFS client versions 1.1.0 and up, or 1.0.2.
- The storage statistics currently count each LFS object multiple times for
every project linking to it.
[reconfigure gitlab]: ../../administration/restart_gitlab.md#omnibus-gitlab-reconfigure "How to reconfigure Omnibus GitLab"
[restart gitlab]: ../../administration/restart_gitlab.md#installations-from-source "How to restart GitLab"
[eep]: https://about.gitlab.com/pricing/ "GitLab Premium"
[ee-2760]: https://gitlab.com/gitlab-org/gitlab/merge_requests/2760
doc/workflow/lfs/manage_large_binaries_with_git_lfs.md
View file @ b734365b
# Git LFS ---
redirect_to: '../../administration/lfs/manage_large_binaries_with_git_lfs.md'
---
Managing large files such as audio, video and graphics files has always been one This document was moved to [another location](../../administration/lfs/manage_large_binaries_with_git_lfs.md).
of the shortcomings of Git. The general recommendation is to not have Git repositories
larger than 1GB to preserve performance.
![Git LFS tracking status](img/lfs-icon.png)
An LFS icon is shown on files tracked by Git LFS to denote if a file is stored
as a blob or as an LFS pointer.
## How it works
Git LFS client talks with the GitLab server over HTTPS. It uses HTTP Basic Authentication
to authorize client requests. Once the request is authorized, Git LFS client receives
instructions from where to fetch or where to push the large file.
## GitLab server configuration
Documentation for GitLab instance administrators is under [LFS administration doc](lfs_administration.md).
## Requirements
- Git LFS is supported in GitLab starting with version 8.2
- Git LFS must be enabled under project settings
- [Git LFS client](https://git-lfs.github.com) version 1.0.1 and up
## Known limitations
- Git LFS v1 original API is not supported since it was deprecated early in LFS
development
- When SSH is set as a remote, Git LFS objects still go through HTTPS
- Any Git LFS request will ask for HTTPS credentials to be provided so a good Git
credentials store is recommended
- Git LFS always assumes HTTPS so if you have GitLab server on HTTP you will have
to add the URL to Git config manually (see [troubleshooting](#troubleshooting))
NOTE: **Note:**
With 8.12 GitLab added LFS support to SSH. The Git LFS communication
still goes over HTTP, but now the SSH client passes the correct credentials
to the Git LFS client, so no action is required by the user.
## Using Git LFS
Lets take a look at the workflow when you need to check large files into your Git
repository with Git LFS. For example, if you want to upload a very large file and
check it into your Git repository:
```bash
git clone git@gitlab.example.com:group/project.git
git lfs install # initialize the Git LFS project
git lfs track "*.iso" # select the file extensions that you want to treat as large files
```
Once a certain file extension is marked for tracking as a LFS object you can use
Git as usual without having to redo the command to track a file with the same extension:
```bash
cp ~/tmp/debian.iso ./ # copy a large file into the current directory
git add . # add the large file to the project
git commit -am "Added Debian iso" # commit the file meta data
git push origin master # sync the git repo and large file to the GitLab server
```
**Make sure** that `.gitattributes` is tracked by Git. Otherwise Git
LFS will not be working properly for people cloning the project:
```bash
git add .gitattributes
```
Cloning the repository works the same as before. Git automatically detects the
LFS-tracked files and clones them via HTTP. If you performed the `git clone`
command with a SSH URL, you have to enter your GitLab credentials for HTTP
authentication.
```bash
git clone git@gitlab.example.com:group/project.git
```
If you already cloned the repository and you want to get the latest LFS object
that are on the remote repository, eg. for a branch from origin:
```bash
git lfs fetch origin master
```
### Migrate an existing repo to Git LFS
Read the documentation on how to [migrate an existing Git repo with Git LFS](../../topics/git/migrate_to_git_lfs/index.md).
## File Locking
> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/issues/35856) in GitLab 10.5.
The first thing to do before using File Locking is to tell Git LFS which
kind of files are lockable. The following command will store PNG files
in LFS and flag them as lockable:
```bash
git lfs track "*.png" --lockable
```
After executing the above command a file named `.gitattributes` will be
created or updated with the following content:
```bash
*.png filter=lfs diff=lfs merge=lfs -text lockable
```
You can also register a file type as lockable without using LFS
(In order to be able to lock/unlock a file you need a remote server that implements the LFS File Locking API),
in order to do that you can edit the `.gitattributes` file manually:
```bash
*.pdf lockable
```
After a file type has been registered as lockable, Git LFS will make
them readonly on the file system automatically. This means you will
need to lock the file before editing it.
### Managing Locked Files
Once you're ready to edit your file you need to lock it first:
```bash
git lfs lock images/banner.png
Locked images/banner.png
```
This will register the file as locked in your name on the server:
```bash
git lfs locks
images/banner.png joe ID:123
```
Once you have pushed your changes, you can unlock the file so others can
also edit it:
```bash
git lfs unlock images/banner.png
```
You can also unlock by id:
```bash
git lfs unlock --id=123
```
If for some reason you need to unlock a file that was not locked by you,
you can use the `--force` flag as long as you have a `maintainer` access on
the project:
```bash
git lfs unlock --id=123 --force
```
## Troubleshooting
### error: Repository or object not found
There are a couple of reasons why this error can occur:
- You don't have permissions to access certain LFS object
Check if you have permissions to push to the project or fetch from the project.
- Project is not allowed to access the LFS object
LFS object you are trying to push to the project or fetch from the project is not
available to the project anymore. Probably the object was removed from the server.
- Local Git repository is using deprecated LFS API
### Invalid status for `<url>` : 501
Git LFS will log the failures into a log file.
To view this log file, while in project directory:
```bash
git lfs logs last
```
If the status `error 501` is shown, it is because:
- Git LFS is not enabled in project settings. Check your project settings and
enable Git LFS.
- Git LFS support is not enabled on the GitLab server. Check with your GitLab
administrator why Git LFS is not enabled on the server. See
[LFS administration documentation](lfs_administration.md) for instructions
on how to enable LFS support.
- Git LFS client version is not supported by GitLab server. Check your Git LFS
version with `git lfs version`. Check the Git config of the project for traces
of deprecated API with `git lfs -l`. If `batch = false` is set in the config,
remove the line and try to update your Git LFS client. Only version 1.0.1 and
newer are supported.
### getsockopt: connection refused
If you push a LFS object to a project and you receive an error similar to:
`Post <URL>/info/lfs/objects/batch: dial tcp IP: getsockopt: connection refused`,
the LFS client is trying to reach GitLab through HTTPS. However, your GitLab
instance is being served on HTTP.
This behaviour is caused by Git LFS using HTTPS connections by default when a
`lfsurl` is not set in the Git config.
To prevent this from happening, set the lfs url in project Git config:
```bash
git config --add lfs.url "http://gitlab.example.com/group/project.git/info/lfs"
```
### Credentials are always required when pushing an object
NOTE: **Note:**
With 8.12 GitLab added LFS support to SSH. The Git LFS communication
still goes over HTTP, but now the SSH client passes the correct credentials
to the Git LFS client, so no action is required by the user.
Given that Git LFS uses HTTP Basic Authentication to authenticate the user pushing
the LFS object on every push for every object, user HTTPS credentials are required.
By default, Git has support for remembering the credentials for each repository
you use. This is described in [Git credentials man pages](https://git-scm.com/docs/gitcredentials).
For example, you can tell Git to remember the password for a period of time in
which you expect to push the objects:
```bash
git config --global credential.helper 'cache --timeout=3600'
```
This will remember the credentials for an hour after which Git operations will
require re-authentication.
If you are using OS X you can use `osxkeychain` to store and encrypt your credentials.
For Windows, you can use `wincred` or Microsoft's [Git Credential Manager for Windows](https://github.com/Microsoft/Git-Credential-Manager-for-Windows/releases).
More details about various methods of storing the user credentials can be found
on [Git Credential Storage documentation](https://git-scm.com/book/en/v2/Git-Tools-Credential-Storage).
### LFS objects are missing on push
GitLab checks files to detect LFS pointers on push. If LFS pointers are detected, GitLab tries to verify that those files already exist in LFS on GitLab.
Verify that LFS in installed locally and consider a manual push with `git lfs push --all`.
If you are storing LFS files outside of GitLab you can disable LFS on the project by setting `lfs_enabled: false` with the [projects API](../../api/projects.md#edit-project).
### Hosting LFS objects externally
It is possible to host LFS objects externally by setting a custom LFS url with `git config -f .lfsconfig lfs.url https://example.com/<project>.git/info/lfs`.
You might choose to do this if you are using an appliance like a Sonatype Nexus to store LFS data. If you choose to use an external LFS store,
GitLab will not be able to verify LFS objects which means that pushes will fail if you have GitLab LFS support enabled.
To stop push failure, LFS support can be disabled in the [Project settings](../../user/project/settings/index.md). This means you will lose GitLab LFS value-adds (Verifying LFS objects, UI integration for LFS).
doc/workflow/lfs/migrate_from_git_annex_to_git_lfs.md
View file @ b734365b
# Migration guide from Git Annex to Git LFS ---
redirect_to: '../../administration/lfs/migrate_from_git_annex_to_git_lfs.md'
>**Note:**
Git Annex support [has been removed][issue-remove-annex] in GitLab Enterprise
Edition 9.0 (2017/03/22).
Both [Git Annex][] and [Git LFS][] are tools to manage large files in Git.
## History
Git Annex [was introduced in GitLab Enterprise Edition 7.8][post-3], at a time
where Git LFS didn't yet exist. A few months later, GitLab brought support for
Git LFS in [GitLab 8.2][post-2] and is available for both Community and
Enterprise editions.
## Differences between Git Annex and Git LFS
Some items below are general differences between the two protocols and some are
ones that GitLab developed.
- Git Annex works only through SSH, whereas Git LFS works both with SSH and HTTPS
(SSH support was added in GitLab 8.12).
- Annex files are stored in a sub-directory of the normal repositories, whereas
LFS files are stored outside of the repositories in a place you can define.
- Git Annex requires a more complex setup, but has much more options than Git
LFS. You can compare the commands each one offers by running `man git-annex`
and `man git-lfs`.
- Annex files cannot be browsed directly in GitLab's interface, whereas LFS
files can.
## Migration steps
>**Note:**
Since Git Annex files are stored in a sub-directory of the normal repositories
(`.git/annex/objects`) and LFS files are stored outside of the repositories,
they are not compatible as they are using a different scheme. Therefore, the
migration has to be done manually per repository.
There are basically two steps you need to take in order to migrate from Git
Annex to Git LFS.
### TL; DR
If you know what you are doing and want to skip the reading, this is what you
need to do (we assume you have [git-annex enabled](../git_annex.md#using-gitlab-git-annex) in your
repository and that you have made backups in case something goes wrong).
Fire up a terminal, navigate to your Git repository and:
1. Disable `git-annex`:
```bash
git annex sync --content
git annex direct
git annex uninit
git annex indirect
```
1. Enable `git-lfs`:
```
git lfs install
git lfs track <files>
git add .
git commit -m "commit message"
git push
```
### Disabling Git Annex in your repo
Before changing anything, make sure you have a backup of your repository first.
There are a couple of ways to do that, but you can simply clone it to another
local path and maybe push it to GitLab if you want a remote backup as well.
Here you'll find a guide on
[how to back up a **git-annex** repository to an external hard drive][bkp-ext-drive].
Since Annex files are stored as objects with symlinks and cannot be directly
modified, we need to first remove those symlinks.
NOTE: **Note:**
Make sure the you read about the [`direct` mode][annex-direct] as it contains
useful information that may fit in your use case. Note that `annex direct` is
deprecated in Git Annex version 6, so you may need to upgrade your repository
if the server also has Git Annex 6 installed. Read more in the
[Git Annex troubleshooting tips](../git_annex.md#troubleshooting-tips) section.
1. Backup your repository
```bash
cd repository
git annex sync --content
cd ..
git clone repository repository-backup
cd repository-backup
git annex get
cd ..
```
1. Use `annex direct`:
```bash
cd repository
git annex direct
```
The output should be similar to this:
```bash
commit
On branch master
Your branch is up-to-date with 'origin/master'.
nothing to commit, working tree clean
ok
direct debian.iso ok
direct ok
```
1. Disable Git Annex with [`annex uninit`][uninit]:
```bash
git annex uninit
```
The output should be similar to this:
```bash
unannex debian.iso ok
Deleted branch git-annex (was 2534d2c).
```
This will `unannex` every file in the repository, leaving the original files.
1. Switch back to `indirect` mode:
```bash
git annex indirect
```
The output should be similar to this:
```bash
(merging origin/git-annex into git-annex...)
(recording state in git...)
commit (recording state in git...)
ok
(recording state in git...)
[master fac3194] commit before switching to indirect mode
1 file changed, 1 deletion(-)
delete mode 120000 alpine-virt-3.4.4-x86_64.iso
ok
indirect ok
ok
```
--- ---
At this point, you have two options. Either add, commit and push the files This document was moved to [another location](../../administration/lfs/migrate_from_git_annex_to_git_lfs.md).
directly back to GitLab or switch to Git LFS. We will tackle the LFS switch in
the next section.
### Enabling Git LFS in your repo
Git LFS is enabled by default on all GitLab products (GitLab CE, GitLab EE,
GitLab.com), therefore, you don't need to do anything server-side.
1. First, make sure you have `git-lfs` installed locally:
```bash
git lfs help
```
If the terminal doesn't prompt you with a full response on `git-lfs` commands,
[install the Git LFS client][install-lfs] first.
1. Inside the repo, run the following command to initiate LFS:
```bash
git lfs install
```
1. Enable `git-lfs` for the group of files you want to track. You
can track specific files, all files containing the same extension, or an
entire directory:
```bash
git lfs track images/01.png # per file
git lfs track **/*.png # per extension
git lfs track images/ # per directory
```
Once you do that, run `git status` and you'll see `.gitattributes` added
to your repo. It collects all file patterns that you chose to track via
`git-lfs`.
1. Add the files, commit and push them to GitLab:
```bash
git add .
git commit -m "commit message"
git push
```
If your remote is set up with HTTP, you will be asked to enter your login
credentials. If you have [2FA enabled](../../user/profile/account/two_factor_authentication.md), make sure to use a
[personal access token](../../user/profile/account/two_factor_authentication.md#personal-access-tokens)
instead of your password.
## Removing the Git Annex branches
After the migration finishes successfully, you can remove all `git-annex`
related branches from your repository.
On GitLab, navigate to your project's **Repository ➔ Branches** and delete all
branches created by Git Annex: `git-annex`, and all under `synced/`.
![repository branches](images/git-annex-branches.png)
You can also do this on the command line with:
```bash
git branch -d synced/master
git branch -d synced/git-annex
git push origin :synced/master
git push origin :synced/git-annex
git push origin :git-annex
git remote prune origin
```
If there are still some Annex objects inside your repository (`.git/annex/`)
or references inside `.git/config`, run `annex uninit` again:
```bash
git annex uninit
```
## Further Reading
- (Blog Post) [Getting Started with Git FLS][post-1]
- (Blog Post) [Announcing LFS Support in GitLab][post-2]
- (Blog Post) [GitLab Annex Solves the Problem of Versioning Large Binaries with Git][post-3]
- (GitLab Docs) [Git Annex](../git_annex.md)
- (GitLab Docs) [Git LFS](manage_large_binaries_with_git_lfs.md)
[annex-direct]: https://git-annex.branchable.com/direct_mode/
[bkp-ext-drive]: https://www.thomas-krenn.com/en/wiki/Git-annex_Repository_on_an_External_Hard_Drive
[Git Annex]: http://git-annex.branchable.com/
[Git LFS]: https://git-lfs.github.com/
[install-lfs]: https://git-lfs.github.com/
[issue-remove-annex]: https://gitlab.com/gitlab-org/gitlab/issues/1648
[lfs-track]: https://about.gitlab.com/blog/2017/01/30/getting-started-with-git-lfs-tutorial/#tracking-files-with-lfs
[post-1]: https://about.gitlab.com/blog/2017/01/30/getting-started-with-git-lfs-tutorial/
[post-2]: https://about.gitlab.com/blog/2015/11/23/announcing-git-lfs-support-in-gitlab/
[post-3]: https://about.gitlab.com/blog/2015/02/17/gitlab-annex-solves-the-problem-of-versioning-large-binaries-with-git/
[uninit]: https://git-annex.branchable.com/git-annex-uninit/
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment
GitLab Nexedi Edition | About GitLab | About Nexedi | 沪ICP备2021021310号-2 | 沪ICP备2021021310号-7