This is to improve RAM usage when parsing references utilizing the
existing `cache` struct.

Reduces memory usage when parsing LSIF file of gitlab-workhorse
from 7MB to 4MB.