diff options
| author | Yury Melnichek <melnichek@gmail.com> | 2012-09-11 01:59:48 +0400 |
|---|---|---|
| committer | Alex Zolotarev <alex@maps.me> | 2015-09-23 01:43:32 +0300 |
| commit | 59ae727bef8a55c543d8891d55d016fd956707cc (patch) | |
| tree | 6b75f1e1d64a56ed5daab78ab21fb5ea3b4ca671 /crawler | |
| parent | 704b2fe733ca8a8f5d680c1cf829a124bf3041bd (diff) | |
[crawler] Crawl more Wikitravel pages.
Diffstat (limited to 'crawler')
| -rwxr-xr-x | crawler/wikitravel-download-lists.sh | 1 |
1 files changed, 1 insertions, 0 deletions
diff --git a/crawler/wikitravel-download-lists.sh b/crawler/wikitravel-download-lists.sh index 16f6c8f764..0b3a0136ec 100755 --- a/crawler/wikitravel-download-lists.sh +++ b/crawler/wikitravel-download-lists.sh @@ -9,6 +9,7 @@ wget $LONGPAGES_URL"&limit=5000&offset=0" -O wikitravel-pages-0.html && slee wget $LONGPAGES_URL"&limit=5000&offset=5000" -O wikitravel-pages-1.html && sleep 10s wget $LONGPAGES_URL"&limit=5000&offset=10000" -O wikitravel-pages-2.html && sleep 10s wget $LONGPAGES_URL"&limit=5000&offset=15000" -O wikitravel-pages-3.html && sleep 10s +wget $LONGPAGES_URL"&limit=5000&offset=20000" -O wikitravel-pages-4.html && sleep 10s # Get all redirects. wget $REDIRECTS_URL"&limit=5000&offset=0" -O wikitravel-redirects-0.html && sleep 10s |