Allow crawler access to api

As GitLab pages do access non-authenticated apis such as api/graphql
for the page to render correctly

changelogs/unreleased/allow-robots-api.yml

+---
+title: Allow crawler access to api
+merge_request: 39520
+author:
+type: fixed

public/robots.txt

@@ -15,7 +15,6 @@
 User-Agent: *
 Disallow: /autocomplete/users
 Disallow: /search
-Disallow: /api
 Disallow: /admin
 Disallow: /profile
 Disallow: /dashboard