Even though we delete all temporary refs after pull, git leaves empty
directories in the place where the refs were - for example if there was
a ref

    dir/ref

and we delete ref `ref`, empty dir/ is still leaved there.

That increasingly hurts next pull performance a lot - before pulling git
wants to scan all local refs, and while doing so it descends into all
directories under refs/.

As after several pulls we can have many such empty directories under
refs/backup/, this scanning can take quite some time: e.g. for
lab.nexedi.com normal pull currently takes ~3 minutes, but after doing
pull ~60 times, it can become as bad as ~10 minutes for one pull. And
all that slowness goes away after cleaning refs/backup/ manually.

/cc https://lab.nexedi.com/lab.nexedi.com/lab.nexedi.com/issues/4