Commit 3ac6054b authored by eric sabelhaus's avatar eric sabelhaus

correct User_agent placement in robots.txt

parent cdf0af66
...@@ -4,13 +4,12 @@ ...@@ -4,13 +4,12 @@
# User-Agent: * # User-Agent: *
# Disallow: / # Disallow: /
User-Agent: *
# Add a 1 second delay between successive requests to the same server, limits resources used by crawler # Add a 1 second delay between successive requests to the same server, limits resources used by crawler
# Only some crawlers respect this setting, e.g. Googlebot does not # Only some crawlers respect this setting, e.g. Googlebot does not
# Crawl-delay: 1 # Crawl-delay: 1
# Based on details in https://gitlab.com/gitlab-org/gitlab-ce/blob/master/config/routes.rb, https://gitlab.com/gitlab-org/gitlab-ce/blob/master/spec/routing, and using application # Based on details in https://gitlab.com/gitlab-org/gitlab-ce/blob/master/config/routes.rb, https://gitlab.com/gitlab-org/gitlab-ce/blob/master/spec/routing, and using application
User-Agent: *
Disallow: /autocomplete/users Disallow: /autocomplete/users
Disallow: /search Disallow: /search
Disallow: /api Disallow: /api
...@@ -23,12 +22,14 @@ Disallow: /groups/*/edit ...@@ -23,12 +22,14 @@ Disallow: /groups/*/edit
Disallow: /users Disallow: /users
# Global snippets # Global snippets
User-Agent: *
Disallow: /s/ Disallow: /s/
Disallow: /snippets/new Disallow: /snippets/new
Disallow: /snippets/*/edit Disallow: /snippets/*/edit
Disallow: /snippets/*/raw Disallow: /snippets/*/raw
# Project details # Project details
User-Agent: *
Disallow: /*/*.git Disallow: /*/*.git
Disallow: /*/*/fork/new Disallow: /*/*/fork/new
Disallow: /*/*/repository/archive* Disallow: /*/*/repository/archive*
......
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment