| Age | Commit message (Collapse) | Author |
|---|
|
Fix incorrect range 'apt-mirror'
|
|
|
|
Fix: added closing bracket
|
|
Paranoid mode: Abort if we downloaded a corrupted file
|
|
88649412fdfbd6648fe87cf5e91a2196d37d488e Invalid if scope set on commit
|
|
|
|
|
|
If a file needs to be updated, we can throw away the local copy
|
|
This prevents wget from making an extra request to the server to check
if the file needs to be downloaded - we already know that we want to
download it because the size is different.
|
|
Support insecure repos
|
|
If a repository does not have a Release file it is insecure as its
integrity cannot be validated with a GPG signature. In that situation
apt would fall back to downloading the well-known Packages and Contents
index files, so we try to mimic this behavior.
|
|
Taking the example from the specs at
https://wiki.debian.org/DebianRepository/Format#Flat_Repository_Format
The correct way to specify a flat repository is by giving a `/` or `.`
for the directory. E.g., taking the example from Stifler6996#2, instead
of trying with
```
deb-amd64 http://ftp.gwdg.de/pub/opensuse/repositories/home:/stbuehler:/lighttpd-1.4.x/xUbuntu_20.04
```
we should be using
```
deb-amd64 http://ftp.gwdg.de/pub/opensuse/repositories/home:/stbuehler:/lighttpd-1.4.x/xUbuntu_20.04 /
```
|
|
This allows to mirror a repo like
```
deb https://nvidia.github.io/nvidia-docker/ubuntu18.04/amd64/ /
```
which has an `InRelease` file, but no `Release` file.
|
|
Revert "The debian network installer needs these files too"
|
|
This reverts commit 6514286140ae3f585729aa273f356ab239d1726e.
This was already addressed once in the past, sorry for not catching it
on time. Quoting the original commit 408721e8f04f3e250744bcfde5b5caf0e864f1e8
```
The standard way to include the installation components is via the
mirror spec line, e.g.x
deb-amd64 http://us.archive.ubuntu.com/ubuntu wily main restricted universe multiverse main/debian-installer
This is a universal syntax, used by everybody in the same way. I think
it would be better not for apt-mirror to fetch things I didn't ask for,
and to simply continue it's current behavior of doing what it is asked
to do.
```
|
|
|
|
|
|
The debian network installer needs these files too
|
|
Steps to reproduce:
virt-install --osinfo debianbullseye --memory 1024 --disk size=10 \
--location http://ftp.debian.org/debian/dists/bullseye/main/installer-amd64/current/images/netboot/debian-installer/amd64/,kernel=linux,initrd=initrd.gz \
The netboot installer will try to:
1. Read the Release file under binary-$arch
2. Download the Packages file under the debian-installer/binary-$arch
3. Download the microdebs which were described in such Package files
|
|
Release is needed for ipxe/netboot. This commit tidies my quick fix to only download Release and not just everything
|
|
Let's skip downloading the non-existent uncompressed Packages file as it
is failing 100% of the time anyway.
It seems that the `Release` file under the binary directory is necessary
in some situations, even though it is documented as unnecessary in
https://wiki.debian.org/DebianRepository/Format#Legacy_per-component-and-architecture_Release_files
> Legacy per-component-and-architecture Release files
> Servers may provide legacy Release files in
> "dists/$DIST/$COMP/binary-$ARCH/Release".
>
> It should contain only the following fields:
>
> - Archive (old name for Suite)
> - Origin
> - Label
> - Acquire-By-Hash (optional)
> - Component: $COMP (singular variant of Components)
> - Architecture: $ARCH (current architecture)
>
> Clients must not use them.
|
|
The files inside /main/binary-amd64 (and others) are all required in order for netboot to work. The "Release" file describes the folders arch and other details.
It does not affect normal mirror operations such as apt update or apt install.
```
Archive: stable
Origin: Debian
Label: Debian
Version: 11.3
Acquire-By-Hash: yes
Component: main
Architecture: amd64
```
We might need more work regarding this problem: https://github.com/chutzimir/apt-mirror/commit/2ac9a69498bbb98dfc5a47092e5c7ec097a3dd91
This commit has been tested and confirmed to work with Ubuntu Focal, which can be netbooted and installed directly from a local mirror. It should therefore work for Debian as well, but I have not tested (would like to test specifically with Bullseye to start with).
|
|
In recent testing (as of this commit) 3 items were unavailable from deb.debian.org/debian (site unreachable: ERR_CONNECTION_RESET):
```
http://deb.debian.org/debian/pool/main/libc/libcorona-perl/libcorona-perl_0.1004-4_all.deb
http://deb.debian.org/debian/pool/main/n/nicovideo-dl/nicovideo-dl_0.0.20190126-1.1_all.deb
http://deb.debian.org/debian/pool/main/r/r-cran-covid19us/r-cran-covid19us_0.1.7-1_all.deb
```
The result is a connection timeout after the default wget timeout of 900 second - per link. Coupled with the '-t 5' (--retries) we had a stall in operation for 1h15m. This was during concurrent threads. if the nthreads was 1 this would have resulted in 3h45m delay in processing.
The solution was to reduce connection timeout to 60 seconds, and reduce the retries to only once.
-T, (--timeout=SECONDS) Same as setting all these to the same timeout: --dns-timeout, --connect-timeout, --read-timeout
-t, (--tries=NUMBER) Set number of retries to NUMBER. 0=unlimited
These values are acceptable with today's internet, and if a file fails during a run, it will get picked up next time around, assuming the problem has been resolved.
This could also be a candidate for a configurable option(s) in the config file.
|
|
|
|
Neither Debian nor Ubuntu mirrors host the "i18n/Index"
|
|
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=649314
To also quote https://github.com/Debian/apt/commit/8e3900d0d7efc11d538b944ed1d9e4e3d5286ff6
```
i18n/Index was never used outside debian - and even here it isn't used
consistently as only 'main' has such a file. As the Release file now
includes the Translation-* files we therefore drop support for
i18n/Index.
A version supporting it was never part of a debian release and still
supporting it would mean that we get 99% of the time a 404 as response
to the request anyway and confuse archive maintainers who want to
provide all files APT tries to acquire.
```
Fixes Stifler6996/apt-mirror#24
|
|
* Reduce copy-paste of sanitise_uri logic
* Don't strip port number in sanitise_uri
|
|
|
|
https://wiki.debian.org/DebianRepository/Format#Acquire-By-Hash
This avoids race conditions where the canonical filenames may change as
we download them.
fixes Stifler6996/apt-mirror#16
|
|
(#22)
This can save significant space as it will link the checksummed files to
the actual files, which is the original intent of these files.
And if the link is not supported or fails - we fall back to the original
file copy.
|
|
https://wiki.debian.org/DebianRepository/Format#Compression_of_indices)
Servers should offer only xz compressed files, except for the
special cases listed above.
even though the Release files would have checksums for the uncompressed
versions as well.
https://wiki.debian.org/DebianRepository/Format#MD5Sum.2C_SHA1.2C_SHA256
The checksum and sizes shall match the actual existing files. If
indexes are compressed, checksum data must be provided for
uncompressed files as well, even if not present on the server.
And if Debian and Ubuntu don't serve the uncompressed ones, do we really
have to bother?
* lzma extension also supported
https://wiki.debian.org/DebianRepository/Format#Compression_of_indices
Unless a compression is indicated by the filename of the indices below, and index may be compressed in one or multiple of the following formats:
- No compression (no extension)
- XZ (.xz extension)
- Gzip (.gz extension, usually for Contents files, and diffs)
- Bzip2 (.bz2 extension, usually for Translations)
- LZMA (.lzma extension)
|
|
* Parse the release files for all relevant metadata in one shot
There is no need to do the same thing over and over. Parse the Release
file once, and pick only the files we need to mirror.
* Arch all is always required
Even when we mirror only a specific arch, the "all" architecture
packages are required for that arch.
Documented at https://wiki.debian.org/DebianRepository/Format#Architectures
Real example: https://apt.puppetlabs.com/dists/focal/Release
deb [arch=amd64] https://apt.puppetlabs.com/ focal puppet
fixes Stifler6996/apt-mirror#14
|
|
Refactor this skipclean logic to only do it in one place. The logic is
pretty simple - if we decided to download a file, then we *must* keep
it.
There is no need for the copy-pasta.
Not keeping the logic to keep the uncompressed version of compressed
files, because if we did not download the uncompressed files, there is
no reason why we should be trying to keep them.
|
|
Spreading the download URLs in a round-robin method spreads "similar"
files to different threads. The point being that "similar" files have
"similar" sizes, so overall each thread's download size varies less.
In this case "similar" files can mean:
- similarly named deb files, which can have "similar" sizes
(e.g., different versions of libreoffice)
- metadata for a specific repo, which can have similar sizes, compared
to that of other repos
While this is not a perfect balance, it is still more balanced compared
to cutting the big list in sequential chunks.
I noticed this when adding a new distro flavor and would end up with a
single wget stuck on downloading all the libreoffice-dev-doc packages
while all the other threads were long finished.
|
|
Allows users to "prettyify" their configs to make them more readable.
fixes #13
|
|
|
|
This will cause wget to send a HEAD request allowing
us to check the timestamp and file size.
Co-authored-by: Daniel Köster <koester@iabg.de>
|
|
during clean function, removing unnecessary directories fails if a file has been left behind.
|
|
|
|
|
|
|
|
|
|
|
|
Add support for basic source repositories that do not have component or distribution sections.
|
|
|
|
Translations are now using xy instead of bz2. Updated to look for both compression formats.
|
|
|
|
This fixes the issue with 64x64@2 Icon files and others that contain the "@" symbol in them.
|
|
|
|
|
