Leave object pools when destroying projects

Closes gitaly#1415

See merge request gitlab-org/gitlab-ce!23869