If you already cloned the repository and you want to get the latest LFS object
that are on the remote repository, eg. for a branch from origin:
```shell
git lfs fetch origin master
```
### Migrate an existing repo to Git LFS
Read the documentation on how to [migrate an existing Git repo with Git LFS](../../topics/git/migrate_to_git_lfs/index.md).
### Removing objects from LFS
To remove objects from LFS:
1. Use [BFG-Cleaner](../../user/project/repository/reducing_the_repo_size_using_git.md#using-the-bfg-repo-cleaner) or [filter-branch](../../user/project/repository/reducing_the_repo_size_using_git.md#using-git-filter-branch) to remove the objects from the repository.
1. Delete the relevant LFS lines for the objects you have removed from your `.gitattributes` file and commit those changes.
## File Locking
> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/issues/35856) in GitLab 10.5.
The first thing to do before using File Locking is to tell Git LFS which
kind of files are lockable. The following command will store PNG files
in LFS and flag them as lockable:
```shell
git lfs track "*.png"--lockable
```
After executing the above command a file named `.gitattributes` will be
This will remember the credentials for an hour after which Git operations will
require re-authentication.
If you are using OS X you can use `osxkeychain` to store and encrypt your credentials.
For Windows, you can use `wincred` or Microsoft's [Git Credential Manager for Windows](https://github.com/Microsoft/Git-Credential-Manager-for-Windows/releases).
More details about various methods of storing the user credentials can be found
on [Git Credential Storage documentation](https://git-scm.com/book/en/v2/Git-Tools-Credential-Storage).
### LFS objects are missing on push
GitLab checks files to detect LFS pointers on push. If LFS pointers are detected, GitLab tries to verify that those files already exist in LFS on GitLab.
Verify that LFS is installed locally and consider a manual push with `git lfs push --all`.
If you are storing LFS files outside of GitLab you can disable LFS on the project by setting `lfs_enabled: false` with the [projects API](../../api/projects.md#edit-project).
### Hosting LFS objects externally
It is possible to host LFS objects externally by setting a custom LFS url with `git config -f .lfsconfig lfs.url https://example.com/<project>.git/info/lfs`.
You might choose to do this if you are using an appliance like a Sonatype Nexus to store LFS data. If you choose to use an external LFS store,
GitLab will not be able to verify LFS objects which means that pushes will fail if you have GitLab LFS support enabled.
To stop push failure, LFS support can be disabled in the [Project settings](../../user/project/settings/index.md). This means you will lose GitLab LFS value-adds (Verifying LFS objects, UI integration for LFS).
This document was moved to [another location](../../topics/git/lfs/index.md).
| [NGINX](#nginx) | Routes requests to appropriate components, terminates SSL | [✅][nginx-omnibus] | [✅][nginx-charts] | [⚙][nginx-charts] | [✅](https://about.gitlab.com/handbook/engineering/infrastructure/production/architecture/#service-architecture) | [⤓][nginx-source] | ❌ | CE & EE |
| [Unicorn (GitLab Rails)](#unicorn) | Handles requests for the web interface and API | [✅][unicorn-omnibus] | [✅][unicorn-charts] | [✅][unicorn-charts] | [✅](../user/gitlab_com/index.md#unicorn) | [⚙][unicorn-source] | [✅][gitlab-yml] | CE & EE |
| [Sidekiq](#sidekiq) | Background jobs processor | [✅][sidekiq-omnibus] | [✅][sidekiq-charts] | [✅](https://docs.gitlab.com/charts/charts/gitlab/sidekiq/index.html) | [✅](../user/gitlab_com/index.md#sidekiq) | [✅][gitlab-yml] | [✅][gitlab-yml] | CE & EE |
| [Gitaly](#gitaly) | Git RPC service for handling all Git calls made by GitLab | [✅][gitaly-omnibus] | [✅][gitaly-charts] | [✅][gitaly-charts] | [✅](https://about.gitlab.com/handbook/engineering/infrastructure/production/architecture/#service-architecture) | [⚙][gitaly-source] | ✅ | CE & EE |
| [Praefect](#praefect) | A transparent proxy between any Git client and Gitaly storage nodes. | [✅][gitaly-omnibus] | [❌][gitaly-charts] | [❌][gitaly-charts] | [✅](https://about.gitlab.com/handbook/engineering/infrastructure/production/architecture/#service-architecture) | [⚙][praefect-source] | ✅ | CE & EE |
| [GitLab Workhorse](#gitlab-workhorse) | Smart reverse proxy, handles large HTTP requests | [✅][workhorse-omnibus] | [✅][workhorse-charts] | [✅][workhorse-charts] | [✅](https://about.gitlab.com/handbook/engineering/infrastructure/production/architecture/#service-architecture) | [⚙][workhorse-source] | ✅ | CE & EE |
| [GitLab Shell](#gitlab-shell) | Handles `git` over SSH sessions | [✅][shell-omnibus] | [✅][shell-charts] | [✅][shell-charts] | [✅](https://about.gitlab.com/handbook/engineering/infrastructure/production/architecture/#service-architecture) | [⚙][shell-source] | [✅][gitlab-yml] | CE & EE |
- GitLab.com: [Monitoring of GitLab.com](https://about.gitlab.com/handbook/engineering/monitoring/)
[Alert manager](https://prometheus.io/docs/alerting/alertmanager/) is a tool provided by Prometheus that _"handles alerts sent by client applications such as the Prometheus server. It takes care of deduplicating, grouping, and routing them to the correct receiver integration such as email, PagerDuty, or OpsGenie. It also takes care of silencing and inhibition of alerts."_ You can read more in [issue #45740](https://gitlab.com/gitlab-org/gitlab-foss/issues/45740) about what we will be alerting on.
Gitaly is a service designed by GitLab to remove our need for NFS for Git storage in distributed deployments of GitLab (think GitLab.com or High Availability Deployments). As of 11.3.0, this service handles all Git level access in GitLab. You can read more about the project [in the project's readme](https://gitlab.com/gitlab-org/gitaly).
- GitLab.com: [Monitoring of GitLab.com](https://about.gitlab.com/handbook/engineering/monitoring/)
GitLab Exporter is a process designed in house that allows us to export metrics about GitLab application internals to Prometheus. You can read more [in the project's readme](https://gitlab.com/gitlab-org/gitlab-exporter).
[GitLab Shell](https://gitlab.com/gitlab-org/gitlab-shell) is a program designed at GitLab to handle SSH-based `git` sessions, and modifies the list of authorized keys. GitLab Shell is not a Unix shell nor a replacement for Bash or Zsh.
[GitLab Workhorse](https://gitlab.com/gitlab-org/gitlab-workhorse) is a program designed at GitLab to help alleviate pressure from Unicorn. You can read more about the [historical reasons for developing](https://about.gitlab.com/blog/2016/04/12/a-brief-history-of-gitlab-workhorse/). It's designed to act as a smart reverse proxy to help speed up GitLab as a whole.
MinIO is an object storage server released under Apache License v2.0. It is compatible with Amazon S3 cloud storage service. It is best suited for storing unstructured data such as photos, videos, log files, backups, and container / VM images. Size of an object can range from a few KBs to a maximum of 5TB.
NGINX has an Ingress port for all HTTP requests and routes them to the appropriate sub-systems within GitLab. We are bundling an unmodified version of the popular open source webserver.
- GitLab.com: [Monitoring of GitLab.com](https://about.gitlab.com/handbook/engineering/monitoring/)
[Node Exporter](https://github.com/prometheus/node_exporter) is a Prometheus tool that gives us metrics on the underlying machine (think CPU/Disk/Load). It's just a packaged version of the common open source offering from the Prometheus project.
- GitLab.com: [Monitoring of GitLab.com](https://about.gitlab.com/handbook/engineering/monitoring/)
[`postgres_exporter`](https://github.com/wrouesnel/postgres_exporter) is the community provided Prometheus exporter that will deliver data about PostgreSQL to Prometheus for use in Grafana Dashboards.
- GitLab.com: [Monitoring of GitLab.com](https://about.gitlab.com/handbook/engineering/monitoring/)
[Redis Exporter](https://github.com/oliver006/redis_exporter) is designed to give specific metrics about the Redis process to Prometheus so that we can graph these metrics in Grafana.
Sentry fundamentally is a service that helps you monitor and fix crashes in realtime.
The server is in Python, but it contains a full API for sending events from any language, in any application.
Sentry fundamentally is a service that helps you monitor and fix crashes in realtime. The server is in Python, but it contains a full API for sending events from any language, in any application.
For monitoring deployed apps, see the [Sentry integration docs](../user/project/operations/error_tracking.md)
Sidekiq is a Ruby background job processor that pulls jobs from the Redis queue and processes them. Background jobs allow GitLab to provide a faster request/response cycle by moving work into the background.
[Unicorn](https://yhbt.net/unicorn/) is a Ruby application server that is used to run the core Rails Application that provides the user facing features in GitLab. Often process output you will see this as `bundle` or `config.ru` depending on the GitLab version.
GitLab provides [GitLab Managed Apps](../user/project/clusters/index.md#installing-applications), a one-click install for various applications which can be added directly to your configured cluster. These applications are needed for Review Apps and deployments when using Auto DevOps. You can install them after you create a cluster.
@@ -740,96 +907,3 @@ Note: It is recommended to log into the `git` user using `sudo -i -u git` or `su
## GitLab.com
We've also detailed [our architecture of GitLab.com](https://about.gitlab.com/handbook/engineering/infrastructure/production/architecture/) but this is probably over the top unless you have millions of users.
In April 2019, Francisco Javier López hosted a [Deep Dive] on GitLab's [Git LFS] implementation to share his domain specific knowledge with anyone who may work in this part of the code base in the future. You can find the [recording on YouTube], and the slides on [Google Slides] and in [PDF]. Everything covered in this deep dive was accurate as of GitLab 11.10, and while specific details may have changed since then, it should still serve as a good introduction.
Based on best practices, you should allow SSH traffic from only a known
host or CIDR block. In that case, change the SSH source to be custom and give
it the IP you want to SSH from.
On the Route 53 dashboard, click **Hosted zones** in the left navigation bar:
1. When done, click **Save**.
1. Select an existing hosted zone or, if you do not already have one for your domain, click **Create Hosted Zone**, enter your domain name, and click **Create**.
1. Click **Create Record Set** and provide the following values:
1.**Name:** Use the domain name (the default value) or enter a subdomain.
1.**Type:** Select **A - IPv4 address**.
1.**Alias Target:** Find the **ELB Classic Load Balancers** section and select the classic load balancer we created earlier.
1.**Routing Policy:** We'll use **Simple** but you can choose a different policy based on your use case.
1.**Evaluate Target Health:** We'll set this to **No** but you can choose to have the load balancer route traffic based on target health.
1. Click **Create**.
1. Update your DNS records with your domain registrar. The steps for doing this vary depending on which registrar you use and is beyond the scope of this guide.
## PostgreSQL with RDS
...
...
@@ -265,7 +283,7 @@ Now that the database is created, let's move on to setting up Redis with ElastiC
ElastiCache is an in-memory hosted caching solution. Redis maintains its own
persistence and is used for certain types of the GitLab application.
To set up Redis:
### Redis Subnet Group
1. Navigate to the ElastiCache dashboard from your AWS console.
1. Go to **Subnet Groups** in the left menu, and create a new subnet group.
...
...
@@ -274,6 +292,18 @@ To set up Redis:
### Create a Redis Security Group
1. Navigate to the EC2 dashboard.
1. Select **Security Groups** from the left menu.
1. Click **Create security group** and fill in the details. Give it a name (we'll use `gitlab-redis-sec-group`),
add a description, and choose the VPC we created previously
1. In the **Inbound rules** section, click **Add rule** and add a **Custom TCP** rule, set port `6379`, and set the "Custom" source as the `gitlab-loadbalancer-sec-group` we created earlier.
1. When done, click **Create security group**.
### Create the Redis Cluster
1. Navigate back to the ElastiCache dashboard.
1. Select **Redis** on the left menu and click **Create** to create a new
Redis cluster. Depending on your load, you can choose whether to enable
cluster mode or not. Even without cluster mode on, you still get the
...
...
@@ -281,8 +311,9 @@ To set up Redis:
not to enable it.
1. In the settings section:
1. Give the cluster a name (`gitlab-redis`) and a description.
1. For the version, select the latest of `3.2` series (e.g., `3.2.10`).
1. Select the node type and the number of replicas.
1. For the version, select the latest of `5.0` series (e.g., `5.0.6`).
1. Leave the port as `6379` since this is what we used in our Redis security group above.
1. Select the node type (at least `cache.t3.medium`, but adjust to your needs) and the number of replicas.
1. In the advanced settings section:
1. Select the multi-AZ auto-failover option.
1. Select the subnet group we created previously.
...
...
@@ -292,7 +323,7 @@ To set up Redis:
1. In the security settings, edit the security groups and choose the
`gitlab-security-group` we had previously created.
`gitlab-redis-sec-group` we had previously created.
1. Leave the rest of the settings to their default values or edit to your liking.
1. When done, click **Create**.
...
...
@@ -307,48 +338,6 @@ source.
Similar to the above, jump to the `gitlab-security-group` group
and add a custom TCP rule for port `6379` accessible within itself.
## Load Balancer
On the EC2 dashboard, look for Load Balancer in the left navigation bar:
1. Click the **Create Load Balancer** button.
1. Choose the **Classic Load Balancer**.
1. Give it a name (we'll use `gitlab-loadbalancer`) and for the **Create LB Inside** option, select `gitlab-vpc` from the dropdown menu.
1. In the **Listeners** section, set HTTP port 80, HTTPS port 443, and TCP port 22 for both load balancer and instance protocols and ports.
1. In the **Select Subnets** section, select both public subnets from the list.
1. Click **Assign Security Groups** and select **Create a new security group**, give it a name
(we'll use `gitlab-loadbalancer-sec-group`) and description, and allow both HTTP and HTTPS traffic
from anywhere (`0.0.0.0/0, ::/0`).
1. Click **Configure Security Settings** and select an SSL/TLS certificate from ACM or upload a certificate to IAM.
1. Click **Configure Health Check** and set up a health check for your EC2 instances.
1. For **Ping Protocol**, select HTTP.
1. For **Ping Port**, enter 80.
1. For **Ping Path**, enter `/explore`. (We use `/explore` as it's a public endpoint that does
not require authorization.)
1. Keep the default **Advanced Details** or adjust them according to your needs.
1. Click **Add EC2 Instances** but, as we don't have any instances to add yet, come back
to your load balancer after creating your GitLab instances and add them.
1. Click **Add Tags** and add any tags you need.
1. Click **Review and Create**, review all your settings, and click **Create** if you're happy.
After the Load Balancer is up and running, you can revisit your Security
Groups to refine the access only through the ELB and any other requirements
you might have.
### Configure DNS for Load Balancer
On the Route 53 dashboard, click **Hosted zones** in the left navigation bar:
1. Select an existing hosted zone or, if you do not already have one for your domain, click **Create Hosted Zone**, enter your domain name, and click **Create**.
1. Click **Create Record Set** and provide the following values:
1.**Name:** Use the domain name (the default value) or enter a subdomain.
1.**Type:** Select **A - IPv4 address**.
1.**Alias Target:** Find the **ELB Classic Load Balancers** section and select the classic load balancer we created earlier.
1.**Routing Policy:** We'll use **Simple** but you can choose a different policy based on your use case.
1.**Evaluate Target Health:** We'll set this to **No** but you can choose to have the load balancer route traffic based on target health.
1. Click **Create**.
1. Update your DNS records with your domain registrar. The steps for doing this vary depending on which registrar you use and is beyond the scope of this guide.
## Setting up Bastion Hosts
Since our GitLab instances will be in private subnets, we need a way to connect to these instances via SSH to make configuration changes, perform upgrades, etc. One way of doing this is via a [bastion host](https://en.wikipedia.org/wiki/Bastion_host), sometimes also referred to as a jump box.
-[git-annex to Git-LFS migration guide](../../administration/lfs/migrate_from_git_annex_to_git_lfs.md)
-[git-annex to Git-LFS migration guide](lfs/migrate_from_git_annex_to_git_lfs.md)
-[Towards a production quality open source Git LFS server](https://about.gitlab.com/blog/2015/08/13/towards-a-production-quality-open-source-git-lfs-server/)
If you already cloned the repository and you want to get the latest LFS object
that are on the remote repository, eg. for a branch from origin:
```shell
git lfs fetch origin master
```
### Migrate an existing repo to Git LFS
Read the documentation on how to [migrate an existing Git repo with Git LFS](migrate_to_git_lfs.md).
### Removing objects from LFS
To remove objects from LFS:
1. Use [BFG-Cleaner](../../../user/project/repository/reducing_the_repo_size_using_git.md#using-the-bfg-repo-cleaner) or [filter-branch](../../../user/project/repository/reducing_the_repo_size_using_git.md#using-git-filter-branch) to remove the objects from the repository.
1. Delete the relevant LFS lines for the objects you have removed from your `.gitattributes` file and commit those changes.
## File Locking
> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/issues/35856) in GitLab 10.5.
The first thing to do before using File Locking is to tell Git LFS which
kind of files are lockable. The following command will store PNG files
in LFS and flag them as lockable:
```shell
git lfs track "*.png"--lockable
```
After executing the above command a file named `.gitattributes` will be
This will remember the credentials for an hour after which Git operations will
require re-authentication.
If you are using OS X you can use `osxkeychain` to store and encrypt your credentials.
For Windows, you can use `wincred` or Microsoft's [Git Credential Manager for Windows](https://github.com/Microsoft/Git-Credential-Manager-for-Windows/releases).
More details about various methods of storing the user credentials can be found
on [Git Credential Storage documentation](https://git-scm.com/book/en/v2/Git-Tools-Credential-Storage).
### LFS objects are missing on push
GitLab checks files to detect LFS pointers on push. If LFS pointers are detected, GitLab tries to verify that those files already exist in LFS on GitLab.
Verify that LFS is installed locally and consider a manual push with `git lfs push --all`.
If you are storing LFS files outside of GitLab you can disable LFS on the project by setting `lfs_enabled: false` with the [projects API](../../../api/projects.md#edit-project).
### Hosting LFS objects externally
It is possible to host LFS objects externally by setting a custom LFS url with `git config -f .lfsconfig lfs.url https://example.com/<project>.git/info/lfs`.
You might choose to do this if you are using an appliance like a Sonatype Nexus to store LFS data. If you choose to use an external LFS store,
GitLab will not be able to verify LFS objects which means that pushes will fail if you have GitLab LFS support enabled.
To stop push failure, LFS support can be disabled in the [Project settings](../../../user/project/settings/index.md). This means you will lose GitLab LFS value-adds (Verifying LFS objects, UI integration for LFS).
- Alternative method to [migrate an existing repo to Git LFS](https://github.com/git-lfs/git-lfs/wiki/Tutorial#migrating-existing-repository-data-to-lfs)
- Alternative method to [migrate an existing repo to Git LFS](https://github.com/git-lfs/git-lfs/wiki/Tutorial#migrating-existing-repository-data-to-lfs)
@@ -170,7 +170,7 @@ Some tickets need specific knowledge or a deep understanding of a particular com
Move on to understanding some of GitLab's more advanced features. You can make use of GitLab.com to understand the features from an end-user perspective and then use your own instance to understand setup and configuration of the feature from an Administrative perspective
- Set up and try [Git LFS](../../administration/lfs/manage_large_binaries_with_git_lfs.md)
- Set up and try [Git LFS](../../topics/git/lfs/index.md)
- Get to know the [GitLab API](../../api/README.md), its capabilities and shortcomings
- Learn how to [migrate from SVN to Git](../../user/project/import/svn.md)
- Set up [GitLab CI/CD](../../ci/quick_start/README.md)
