gitlab.com/gitlab-org/gitlab-foss.git - Unnamed repository; edit this file 'description' to name the repository.

Age	Commit message (Collapse)	Author
2018-11-30	Update robots.txt to exclude group_members and project_members, which can ↵	otheus
	expose sensitive user information to the web. Please see https://developers.google.com/search/reference/robots_txt for the correct wildcard format.
2018-09-07	Update robots.txt (#51167)	Moritz Schlarb

2018-03-22	Add the /help page in robots.txt	Achilleas Pipinellis
	The /help page has docs which we don't want to be crawled as we prefer the docs website instead. Related https://gitlab.com/gitlab-org/gitlab-ce/issues/44433
2017-01-18	correct User_agent placement in robots.txt	eric sabelhaus

2016-09-23	update robots.txt disallow	Matt Harrison
	Allows projects in groups starting with "s" while still disallowing the snippets short urls.
2016-05-16	Disallow search engines from indexing uploads from a GitLab project.	Connor Shea
	This can sometimes include sensitive information from private projects and confidential issues. It shouldn't be indexed. Resolves #15551.
2015-10-05	allow crawling of commit page but not patch/diffs	Ben Bodenmiller
	Commit page has valuable information that search engines should be allowed to crawl however the .patch and .diff pages have no new information that is not on commit page
2015-08-18	disallow irrelevant pages by default in robots	Ben Bodenmiller
	Update default robots.txt rules to disallow irrelevant pages that search engines should not care about. This will still allow important pages like the files, commit details, merge requests, issues, comments, etc. to be crawled.
2011-10-09	init commit	gitlabhq