diff options
| author | Giuseppe Pignataro <rogepix@gmail.com> | 2019-02-03 14:30:44 +0300 |
|---|---|---|
| committer | Giuseppe Pignataro <rogepix@gmail.com> | 2019-02-03 14:30:44 +0300 |
| commit | a8a6c91421511e3e0c5669ec0ffff0505f61da70 (patch) | |
| tree | df5dc0545bf08ed1cd8cb43c44bd76ca24a3fc4d /layouts/robots.txt | |
| parent | 1b3dc687b8bb7a6bc348f07aca1b64af93aff24b (diff) | |
Initial Commit
Diffstat (limited to 'layouts/robots.txt')
| -rw-r--r-- | layouts/robots.txt | 66 |
1 files changed, 66 insertions, 0 deletions
diff --git a/layouts/robots.txt b/layouts/robots.txt new file mode 100644 index 0000000..bf731bb --- /dev/null +++ b/layouts/robots.txt @@ -0,0 +1,66 @@ +User-agent: * + +Disallow: /images/ +Disallow: /js/ +Disallow: /css/ + +User-agent: MJ12bot +Disallow: / + +User-agent: AhrefsBot +Disallow: / + +User-agent: BLEXBot +Disallow: / + +# Block SISTRIX +User-agent: SISTRIX Crawler +Disallow: / +User-agent: sistrix +Disallow: / +User-agent: 007ac9 +Disallow: / +User-agent: 007ac9 Crawler +Disallow: / + +# Block Uptime robot +User-agent: UptimeRobot/2.0 +Disallow: / + +# Block Ezooms Robot +User-agent: Ezooms Robot +Disallow: / + +# Block Perl LWP +User-agent: Perl LWP +Disallow: / + +# Block netEstate NE Crawler (+http://www.website-datenbank.de/) +User-agent: netEstate NE Crawler (+http://www.website-datenbank.de/) +Disallow: / + +# Block WiseGuys Robot +User-agent: WiseGuys Robot +Disallow: / + +# Block Turnitin Robot +User-agent: Turnitin Robot +Disallow: / + +# Block Heritrix +User-agent: Heritrix +Disallow: / + +# Block pricepi +User-agent: pimonster +Disallow: / +User-agent: Pimonster +Disallow: / + +User-agent: SurdotlyBot +Disallow: / + +User-agent: ZoominfoBot +Disallow: / + +Sitemap: {{ "sitemap.xml" | absLangURL }}
\ No newline at end of file |