Commit 6a887b20 authored by Will Chandler's avatar Will Chandler Committed by Evan Read

Docs: Add Rugged GVL issue and rbspy to Sidekiq troubleshooting docs

parent c7162942
......@@ -85,6 +85,27 @@ several `WARN` level messages. Here's an example of a single thread's backtrace:
In some cases Sidekiq may be hung and unable to respond to the `TTIN` signal.
Move on to other troubleshooting methods if this happens.
## Ruby profiling with `rbspy`
[rbspy](https://rbspy.github.io) is an easy to use and low-overhead Ruby profiler that can be used to create
flamegraph-style diagrams of CPU usage by Ruby processes.
No changes to GitLab are required to use it and it has no dependencies. To install it:
1. Download the binary from the [`rbspy` releases page](https://github.com/rbspy/rbspy/releases).
1. Make the binary executable.
To profile a Sidekiq worker for one minute, run:
```shell
sudo ./rbspy record --pid <sidekiq_pid> --duration 60 --file /tmp/sidekiq_profile.svg
```
![Example rbspy flamegraph](img/sidekiq_flamegraph.png)
In this example of a flamegraph generated by `rbspy`, almost all of the Sidekiq process's time is spent in `rev_parse`, a native C
function in Rugged. In the stack, we can see `rev_parse` is being called by the `ExpirePipelineCacheWorker`.
## Process profiling with `perf`
Linux has a process profiling tool called `perf` that is helpful when a certain
......@@ -358,3 +379,17 @@ has number of drawbacks, as mentioned in [Why Ruby's Timeout is dangerous (and T
> - in any of your code, regardless of whether it could have possibly raised an exception before
>
> Nobody writes code to defend against an exception being raised on literally any line. That's not even possible. So Thread.raise is basically like a sneak attack on your code that could result in almost anything. It would probably be okay if it were pure-functional code that did not modify any state. But this is Ruby, so that's unlikely :)
## Disable Rugged
Calls into Rugged, Ruby bindings for `libgit2`, [lock the Sidekiq processes's GVL](https://silverhammermba.github.io/emberb/c/#c-in-ruby-threads),
blocking all jobs on that worker from proceeding. If Rugged calls performed by Sidekiq are slow, this can cause significant delays in
background task processing.
By default, Rugged is used when Git repository data is stored on local storage or on an NFS mount.
[Using Rugged is recommened when using NFS](../nfs.md#improving-nfs-performance-with-gitlab), but if
you are using local storage, disabling Rugged can improve Sidekiq performance:
```shell
sudo gitlab-rake gitlab:features:disable_rugged
```
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment