The port number is meaningless when the X-Forwarded-For header
is used, so let's just remove it.

Closes https://gitlab.com/gitlab-org/gitlab-workhorse/issues/201