Remove unused classes on initial Ci::Ref implementation

Closes #216531

See merge request gitlab-org/gitlab!41077