| Age | Commit message (Collapse) | Author |
|---|
|
Resume mirroring after an interrupted state
|
|
This saves wget from making an extra request to the server to check if
the file needs to be downloaded - we already checked that we want to
download it.
|
|
If wget gets interrupted the currently downloading file will be
incomplete, and will have a very recent timestamp.
In this situation, wget will not re-download the file since it will
appear to be more recent than the file it is trying to download.
"--no-if-modified-since" works around this problem by ignoring the
timestamp of the local file.
|
|
Respect the "Acquire-By-Hash" field if present (download `by-hash`)
|
|
If a repository does not have a Release file it is insecure as its
integrity cannot be validated with a GPG signature. In that situation
apt would fall back to downloading the well-known index files, so we try
to mimic this behavior.
|
|
- Support all the documented hashsums when parsing for metadata
- If downloading by-hash, only get the strongest sum (per the spec)
- Provide all the availables hashes in the local mirror
The above are all in line with the spec at
https://wiki.debian.org/DebianRepository/Format
This makes the mirroring of the following flat repo possible:
deb-amd64 https://nvidia.github.io/nvidia-docker/ubuntu18.04/amd64 /
|
|
These regexes were getting a bit confusing so split the logic into four
explicit cases based on:
- Repository format is either flat or debian (i.e., has components or not)
- Architecture is "source" or something else (binary)
All four cases have unique file paths to consider per
https://wiki.debian.org/DebianRepository/Format.
Specifically, flat repositories only support:
- Packages
- Sources
- InRelease
- Release
There is also a specific mention that Translations and Contents indices
are not defined for flat repositories.
|
|
For a consistent mirror we should download the index files from the
`by-hash` directories. This was nicely documented at
https://www.chiark.greenend.org.uk/~cjwatson/blog/no-more-hash-sum-mismatch-errors.html
and also at
https://wiki.debian.org/DebianRepository/Format#Acquire-By-Hash
This solves two problems:
1. Avoids inconsistent mirrors if the contents of the canonical files
get updated while we are doing the mirroring.
2. Download and serves the content of the by-hash directories
|
|
Do the skipclean logic in only one place. The logic is pretty simple -
if we downloaded a file, then we *must* keep it, so it is logical to add
it to `add_url_to_download`.
Also dropping the logic to keep the uncompressed index files as per the
previous commit we do not need to host them.
|
|
I cannot find a single repository serving the uncompressed index files.
Furthermore, per the documentation at
https://wiki.debian.org/DebianRepository/Format#Compression_of_indices
Servers should offer only xz compressed files, except for the
special cases listed above.
Therefore we should only download the compressed files, and only serve
the compressed files, and all clients are expected to understand them.
The uncompressed versions are documented in the Release file, but we can
safely ignore them and only get the compressed ones instead.
|
|
Even though we only mirror specific arch, we need to include the "all"
architecture as well.
Documented at https://wiki.debian.org/DebianRepository/Format#Architectures
Example of a repository that needs this change:
deb [arch=amd64] https://apt.puppetlabs.com/ focal puppet
|
|
There is no need to do the same thing over and over. Parse the Release
file once, and pick only the files that match the architectures and
components we need, plus the extra content necessary for translation,
dep11, cnf, etc.
This would make the following improvements much less painful.
|
|
HEAD
* 'no-i18n-index' of https://github.com/chutzimir/apt-mirror:
Drop support for i18n/Index
|
|
Support for xz translations
|
|
Don't strip port number in sanitise_uri
|
|
Download the by-hash directories
|
|
- In the case of bullseye-security distributed by the debian security camp,
the extension of Translation-en file of bullseye-security is xz format.
(http://security.debian.org/debian-security/dists/bullseye-security/main/i18n/Translation-en.xz)
- Accordingly, the apt-mirror script was modified to support the xz extension.
|
|
Replace "rmdir" with "rm -r" when cleaning directories
|
|
Update the link
|
|
The link points to the following text:
```
There isn't a GitHub Pages site here.
Did you mean to visit apt-mirror.github.io? Please note that this site belongs to a GitHub user and is not an official GitHub site.
If you're the owner of this site, please update your links to use apt-mirror.github.io instead. Subdomains of github.com are deprecated for GitHub Pages. They will not redirect to github.io after April 15, 2021.
```
|
|
This is a follow up of Debian bug #649314
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=649314
To also quote https://github.com/Debian/apt/commit/8e3900d0d7efc11d538b944ed1d9e4e3d5286ff6 (from 2012)
```
i18n/Index was never used outside debian - and even here it isn't used
consistently as only 'main' has such a file. As the Release file now
includes the Translation-* files we therefore drop support for
i18n/Index.
A version supporting it was never part of a debian release and still
supporting it would mean that we get 99% of the time a 404 as response
to the request anyway and confuse archive maintainers who want to
provide all files APT tries to acquire.
```
Fixes apt-mirror/apt-mirror#154
|
|
remove use of progress var
|
|
Support the by-hash/SHA256 dirs
|
|
|
|
Signed-off-by: Benjamin Drung <benjamin.drung@cloud.ionos.com>
|
|
rmdir errors out if directories are nonempty. rm -r is more dependable and gets rid of them regardless
|
|
sanitize_uri is used to translate URIs to pathnames in order to find
downloaded index files.
However, it had been stripping off the port number, while wget creates paths
that include the port number. This means that if your URI includes a port
number, like http://example.com:8080/debian, wget would download files into a
directory named .../skel/example.com:8080/debian/, but apt-mirror would look
for the files in .../skel/example.com/debian/.
It looks like wget has done this for many years, and there's no easy way to
disable it. You can have wget apply its own sanitization filters for
different platforms with --restrict-file-names but Unix mode includes the port
with a ":", and Windows mode replaces the ":" but also replaces other
characters which would mean adjusting sanitise_uri to match. It seems like the
simplest way to be consistent with wget is to remove the port number filter
from sanitise_ui, since ":" is a valid component of a pathname and doesn't
seem to cause any issues.
Fixes #19
|
|
|
|
The DEP11 Components-$arch.yml could be compressed by xz instead of gz.
Thus download Components-$arch.yml files that are compressed with gz,
bz2, or xz.
fixes #69
|
|
Instead of just writing a MD5 file containing the md5 checksums, also
write SHA1 and SHA256 files. These files could be used for sha1sum and
sha256sum.
|
|
A repository without MD5sum will cause warnings:
Use of uninitialized value $lines{"MD5sum:"} in concatenation (.) or
string at /usr/bin/apt-mirror line 787
The written MD5 file is not used by apt-mirror. Thus skip missing MD5Sum
values.
Example repository without MD5Sum:
deb-amd64 https://packages.chef.io/stable-apt xenial main
fixes #66
|
|
When no options are specified for a deb line in the config (e.g. no
[arch=amd64] or similar), apt-mirror will print a warning:
Use of uninitialized value $config{"options"} in pattern match (m//) at
apt-mirror line 300, <CONFIG> line 6.
fixes #68
|
|
Add limit_rate with its default value to the example mirror.list.
Closes #72
|
|
|
|
* Add support for quoted variables with spaces
* Add new function quoted_path to quote and escape the path.
* Change the system() call to system COMMAND LIST to handle
the spaces in path or argument cases.
fixes #43, #67
Signed-off-by: Tao Wang <twang2218@gmail.com>
Signed-off-by: Benjamin Drung <benjamin.drung@profitbricks.com>
|
|
apt-mirror fails when the base_path does not exist. Create the
directories including their parents by replacing mkdir and mkpath with
make_path.
Signed-off-by: Tao Wang <twang2218@gmail.com>
Signed-off-by: Benjamin Drung <benjamin.drung@profitbricks.com>
|
|
fixes #32, #65
Signed-off-by: Tao Wang <twang2218@gmail.com>
Signed-off-by: Benjamin Drung <benjamin.drung@profitbricks.com>
|
|
|
|
Test that the config file is readable (ie. "-e") before dying, not
strictly a file ("-f").
This allows one to more-easily generate dynamic configuration files
and pass them via `apt-mirror /dev/stdin` rather than having to create
temporary files.
Closes: #824493
|
|
apt-mirror hardcodes .gz usage and as such is no longer able to
actually mirror Debian (experimental).
Use .xz or .bz2 index files when there are not .gz index files.
fixes #30
Closes: #819974
Signed-off-by: Benjamin Drung <benjamin.drung@profitbricks.com>
|
|
This reverts commit 591a52e67a40e8fee2e446de7c151845a03268a9.
The standard way to include the installation components is via the
mirror spec line, e.g.x
deb-amd64 http://us.archive.ubuntu.com/ubuntu wily main restricted universe multiverse main/debian-installer
This is a universal syntax, used by everybody in the same way. I think
it would be better not for apt-mirror to fetch things I didn't ask for,
and to simply continue it's current behavior of doing what it is asked
to do.
fixes #64
|
|
Look in the Release file to select files relevant only for selected
architectures.
fixes #56
LP: #1550852
Signed-off-by: Benjamin Drung <benjamin.drung@profitbricks.com>
|
|
fixes #60
Signed-off-by: Benjamin Drung <benjamin.drung@profitbricks.com>
|
|
Include debian-installer components into the mirroring.
|
|
Thus, the mirrored repository can be used for a network installation.
|
|
if fork() fails, it returns undef
|
|
Use 'dpkg --print-architecture' instead of deprecated --print-installation-architecture
|
|
dpkg: warning: obsolete option '--print-installation-architecture', please use '--print-architecture' instead.
|
|
if fork() fails, it returns undef not a negative number
|
|
|
