Commit 2b34fba1 authored by Achilleas Pipinellis's avatar Achilleas Pipinellis

Merge branch 'docs/git-filter-branch' into 'master'

Document how to use filter-branch to reduce the repo file size

Closes #1270

See merge request !914
parents d2f70576 5508f478
...@@ -6,7 +6,7 @@ ...@@ -6,7 +6,7 @@
Repositories within your GitLab instance can grow quickly, especially if you are Repositories within your GitLab instance can grow quickly, especially if you are
using LFS. Their size can grow exponentially and eat up your storage device quite using LFS. Their size can grow exponentially and eat up your storage device quite
quickly. fast.
In order to avoid this from happening, you can set a hard limit for your In order to avoid this from happening, you can set a hard limit for your
repositories' size. This limit can be set globally, per group, or per project, repositories' size. This limit can be set globally, per group, or per project,
...@@ -19,24 +19,14 @@ These settings can be found within each project's settings, in a group's ...@@ -19,24 +19,14 @@ These settings can be found within each project's settings, in a group's
settings and in the Application Settings area for the global value settings and in the Application Settings area for the global value
(`/admin/application_settings`). (`/admin/application_settings`).
### Repository size restrictions The very first push of a new project cannot be checked for size as of now, so
the first push will allow you to upload more than the limit dictates, but every
subsequent push will be denied. LFS objects, however, can be checked on first
push and **will** be rejected if the sum of their sizes exceeds the maximum
allowed repository size.
When a project has reached its size limit, you will not be able to push to it, For more manually purging the files, read the docs on
create a new merge request, or merge existing ones. You will still be able to [reducing the repository size using Git][repo-size].
create new issues, and clone the project though.
Uploading LFS objects will also be denied.
In order to lift these restrictions, the administrator of the GitLab instance
needs to increase the limit on the particular project that exceeded it.
### Current limitations for the repository size check
The first push of a new project cannot be checked for size as of now, so the first
push will allow you to upload more than the limit dictates, but every subsequent
push will be denied.
LFS objects, however, can be checked on first push and **will** be rejected if the
sum of their sizes exceeds the maximum allowed repository size.
[ee-740]: https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/740 [ee-740]: https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/740
[repo-size]: ../../project/repository/reducing_the_repo_size_using_git.md
# Reducing the repository size using Git
A GitLab administrator can set a [repository size limit][admin-repo-size] which
will prevent you to exceed it.
When a project has reached its size limit, you will not be able to push to it,
create a new merge request, or merge existing ones. You will still be able to
create new issues, and clone the project though. Uploading LFS objects will
also be denied.
In order to lift these restrictions, the administrator of the GitLab instance
needs to increase the limit on the particular project that exceeded it or you
need to instruct Git to rewrite changes.
If you exceed the repository size limit, your first thought might be to remove
some data, make a new commit and push back to the repository. Unfortunately,
it's not so easy and that workflow won't work. Deleting files in a commit doesn't
actually reduce the size of the repo since the earlier commits and blobs are
still around. What you need to do is rewrite history with Git's
[`filter-branch` option][gitscm].
Note that even with that method, until `git gc` runs on the GitLab side, the
"removed" commits and blobs will still be around. And if a commit was ever
included in an MR, or if a build was run for a commit, or if a user commented
on it, it will be kept around too. So, in these cases the size will not decrease.
The only fool proof way to actually decrease the repository size is to prune all
the unneeded stuff locally, and then create a new project on GitLab and start
using that instead.
With that being said, you can try reducing your repository size with the
following method.
## Using `git filter-branch` to purge files
>
**Warning:**
Make sure to first make a copy of your repository since rewriting history will
purge the files and information you are about to delete. Also make sure to
inform any collaborators to not use `pull` after your changes, but use `rebase`.
1. Navigate to your repository:
```
cd my_repository/
```
1. Change to the branch you want to remove the big file from:
```
git checkout master
```
1. Use `filter-branch` to remove the big file:
```
git filter-branch --force --tree-filter 'rm -f path/to/big_file.mpg' HEAD
```
1. Instruct Git to purge the unwanted data:
```
git reflog expire --expire=now --all && git gc --prune=now --aggressive
```
1. Lastly, force push to the repository:
```
git push --force origin master
```
Your repository should now be below the size limit.
>**Note:**
As an alternative to `filter-branch`, you can use the `bfg` tool with a
command like: `bfg --delete-files path/to/big_file.mpg`. Read the
[BFG Repo-Cleaner][bfg] documentation for more information.
[admin-repo-size]: ../../admin_area/settings/account_and_limit_settings.md#repository-size-limit
[bfg]: https://rtyley.github.io/bfg-repo-cleaner/
[gitscm]: https://git-scm.com/book/en/v2/Git-Tools-Rewriting-History#The-Nuclear-Option:-filter-branch
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment