Age | Commit message (Collapse) | Author |
|
|
|
The /help page has docs which we don't want to be crawled
as we prefer the docs website instead.
Related https://gitlab.com/gitlab-org/gitlab-ce/issues/44433
|
|
|
|
Allows projects in groups starting with "s" while still disallowing the
snippets short urls.
|
|
This can sometimes include sensitive information from private projects and confidential issues. It shouldn't be indexed. Resolves #15551.
|
|
Commit page has valuable information that search engines should be allowed to crawl however the .patch and .diff pages have no new information that is not on commit page
|
|
Update default robots.txt rules to disallow irrelevant pages that search
engines should not care about. This will still allow important pages
like the files, commit details, merge requests, issues, comments, etc.
to be crawled.
|
|
|