diff options
| author | Marcia Ramos <virtua.creative@gmail.com> | 2018-04-19 23:43:33 +0300 |
|---|---|---|
| committer | Marcia Ramos <virtua.creative@gmail.com> | 2018-04-19 23:43:33 +0300 |
| commit | f3d4f10675fa4c079c259a4c40371f39fcca8ecb (patch) | |
| tree | f83d2b4f0a9c2bfc3d486231a77286026ba1436c /content/robots.txt.erb | |
| parent | 8e1ec002926c75405c8e32f71c7bba6efdf3e49b (diff) | |
fix robots.txt
- Use the full URL, not relative path
- Disallow /ce/ to avoid duplicate content
Diffstat (limited to 'content/robots.txt.erb')
| -rw-r--r-- | content/robots.txt.erb | 8 |
1 files changed, 5 insertions, 3 deletions
diff --git a/content/robots.txt.erb b/content/robots.txt.erb index b4a4b64f..e228e9f8 100644 --- a/content/robots.txt.erb +++ b/content/robots.txt.erb @@ -3,9 +3,11 @@ is_hidden: true --- <% if ENV['CI_COMMIT_REF_NAME'] == 'master' %> -Sitemap: <%= @items['/sitemap.*'].path %> +sitemap: https://docs.gitlab.com/sitemap.xml +user-agent: * +disallow: /ce/ <% else %> # Ban all spiders from the entire site when not on master branch -User-Agent: * -Disallow: / +user-agent: * +disallow: / <% end %> |