So far we had only tested this with Sidekiq.
This change makes sure that WebExporter and
the puma metrics directory can be used.

This is a prerequisite for extracting WebExporter
into a separate process.